2023-11-24T05:50:11.054 INFO:root:teuthology version: 0.0.1.dev205+g2442239 2023-11-24T05:50:11.056 DEBUG:teuthology.run:Teuthology command: teuthology --description fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} --name rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi --owner scheduled_rishabh@teuthology --verbose --archive /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547 -- /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547/orig.config.yaml 2023-11-24T05:50:11.115 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-24T05:50:11.257 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547 branch: rishabh-mgr-vol-misc description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} email: null first_in_suite: false job_id: '7465547' kernel: client: sha1: distro flavor: default kdb: true sha1: 8247bbb5b470f7e2a1f3eefcc16e24cf1338f606 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: rishabh-mgr-vol-misc ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: a13ea5ff00cb4df7a9cc941166ddeb358881144c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: a13ea5ff00cb4df7a9cc941166ddeb358881144c kclient: syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-mgr-vol-misc sha1: a13ea5ff00cb4df7a9cc941166ddeb358881144c owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 3305 sha1: a13ea5ff00cb4df7a9cc941166ddeb358881144c sleep_before_teardown: 0 suite: fs:volumes suite_branch: rishabh-mgr-vol-misc suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: a13ea5ff00cb4df7a9cc941166ddeb358881144c syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi091.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi094.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 2442239d2653456406c25ae0c71b689c8f2657b6 timestamp: 2023-11-24_05:41:33 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-11-24T05:50:11.257 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa; will attempt to use it 2023-11-24T05:50:11.258 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa/tasks 2023-11-24T05:50:11.258 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-24T05:50:11.276 INFO:teuthology.task.internal:Checking packages... 2023-11-24T05:50:11.316 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'a13ea5ff00cb4df7a9cc941166ddeb358881144c' 2023-11-24T05:50:11.316 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-24T05:50:11.316 INFO:teuthology.packaging:ref: None 2023-11-24T05:50:11.316 INFO:teuthology.packaging:tag: None 2023-11-24T05:50:11.316 INFO:teuthology.packaging:branch: rishabh-mgr-vol-misc 2023-11-24T05:50:11.316 INFO:teuthology.packaging:sha1: a13ea5ff00cb4df7a9cc941166ddeb358881144c 2023-11-24T05:50:11.317 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=rishabh-mgr-vol-misc 2023-11-24T05:50:11.466 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7484.ga13ea5ff 2023-11-24T05:50:11.466 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-24T05:50:11.472 INFO:teuthology.task.internal:no buildpackages task found 2023-11-24T05:50:11.473 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-24T05:50:11.478 INFO:teuthology.task.internal:Saving configuration 2023-11-24T05:50:11.499 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-24T05:50:11.507 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-24T05:50:11.533 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi091.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-24 05:45:14.982474', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-24T05:50:11.567 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-24 05:45:14.983959', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-24T05:50:11.568 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-24T05:50:11.573 INFO:teuthology.task.internal:roles: ubuntu@smithi091.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-11-24T05:50:11.573 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-11-24T05:50:11.573 INFO:teuthology.run_tasks:Running task console_log... 2023-11-24T05:50:11.781 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8c555c7b80>, signals=[15]) 2023-11-24T05:50:11.781 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-24T05:50:11.787 INFO:teuthology.task.internal:Opening connections... 2023-11-24T05:50:11.788 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi091.front.sepia.ceph.com 2023-11-24T05:50:11.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi091.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T05:50:11.858 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi094.front.sepia.ceph.com 2023-11-24T05:50:11.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T05:50:11.929 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-24T05:50:11.937 DEBUG:teuthology.orchestra.run.smithi091:> uname -m 2023-11-24T05:50:11.957 INFO:teuthology.orchestra.run.smithi091.stdout:x86_64 2023-11-24T05:50:11.958 DEBUG:teuthology.orchestra.run.smithi091:> cat /etc/os-release 2023-11-24T05:50:12.019 INFO:teuthology.orchestra.run.smithi091.stdout:NAME="Red Hat Enterprise Linux" 2023-11-24T05:50:12.019 INFO:teuthology.orchestra.run.smithi091.stdout:VERSION="8.6 (Ootpa)" 2023-11-24T05:50:12.019 INFO:teuthology.orchestra.run.smithi091.stdout:ID="rhel" 2023-11-24T05:50:12.019 INFO:teuthology.orchestra.run.smithi091.stdout:ID_LIKE="fedora" 2023-11-24T05:50:12.019 INFO:teuthology.orchestra.run.smithi091.stdout:VERSION_ID="8.6" 2023-11-24T05:50:12.019 INFO:teuthology.orchestra.run.smithi091.stdout:PLATFORM_ID="platform:el8" 2023-11-24T05:50:12.019 INFO:teuthology.orchestra.run.smithi091.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-24T05:50:12.019 INFO:teuthology.orchestra.run.smithi091.stdout:ANSI_COLOR="0;31" 2023-11-24T05:50:12.019 INFO:teuthology.orchestra.run.smithi091.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-24T05:50:12.019 INFO:teuthology.orchestra.run.smithi091.stdout:HOME_URL="https://www.redhat.com/" 2023-11-24T05:50:12.019 INFO:teuthology.orchestra.run.smithi091.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-24T05:50:12.020 INFO:teuthology.orchestra.run.smithi091.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-24T05:50:12.020 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T05:50:12.020 INFO:teuthology.orchestra.run.smithi091.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-24T05:50:12.020 INFO:teuthology.orchestra.run.smithi091.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-24T05:50:12.020 INFO:teuthology.orchestra.run.smithi091.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-24T05:50:12.020 INFO:teuthology.orchestra.run.smithi091.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-24T05:50:12.021 INFO:teuthology.lock.ops:Updating smithi091.front.sepia.ceph.com on lock server 2023-11-24T05:50:12.043 DEBUG:teuthology.orchestra.run.smithi094:> uname -m 2023-11-24T05:50:12.065 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2023-11-24T05:50:12.066 DEBUG:teuthology.orchestra.run.smithi094:> cat /etc/os-release 2023-11-24T05:50:12.125 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="Red Hat Enterprise Linux" 2023-11-24T05:50:12.126 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="8.6 (Ootpa)" 2023-11-24T05:50:12.126 INFO:teuthology.orchestra.run.smithi094.stdout:ID="rhel" 2023-11-24T05:50:12.126 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="fedora" 2023-11-24T05:50:12.126 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="8.6" 2023-11-24T05:50:12.126 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el8" 2023-11-24T05:50:12.126 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-24T05:50:12.126 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2023-11-24T05:50:12.126 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-24T05:50:12.126 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://www.redhat.com/" 2023-11-24T05:50:12.127 INFO:teuthology.orchestra.run.smithi094.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-24T05:50:12.127 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-24T05:50:12.127 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T05:50:12.127 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-24T05:50:12.127 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-24T05:50:12.127 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-24T05:50:12.127 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-24T05:50:12.127 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2023-11-24T05:50:12.151 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-24T05:50:12.160 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-24T05:50:12.166 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-24T05:50:12.167 DEBUG:teuthology.orchestra.run.smithi091:> test '!' -e /home/ubuntu/cephtest 2023-11-24T05:50:12.169 DEBUG:teuthology.orchestra.run.smithi094:> test '!' -e /home/ubuntu/cephtest 2023-11-24T05:50:12.189 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-24T05:50:12.196 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-24T05:50:12.197 DEBUG:teuthology.orchestra.run.smithi091:> test -z $(ls -A /var/lib/ceph) 2023-11-24T05:50:12.232 DEBUG:teuthology.orchestra.run.smithi094:> test -z $(ls -A /var/lib/ceph) 2023-11-24T05:50:12.266 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-24T05:50:12.325 INFO:teuthology.run_tasks:Running task kernel... 2023-11-24T05:50:12.341 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'} 2023-11-24T05:50:12.341 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2023-11-24T05:50:12.342 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-11-24T05:50:12.343 DEBUG:teuthology.orchestra.run.smithi091:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-24T05:50:12.345 DEBUG:teuthology.orchestra.run.smithi094:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-24T05:50:12.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T05:50:12.363 DEBUG:teuthology.orchestra.run.smithi094:> uname -r 2023-11-24T05:50:12.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T05:50:12.364 DEBUG:teuthology.orchestra.run.smithi091:> uname -r 2023-11-24T05:50:12.423 INFO:teuthology.orchestra.run.smithi091.stdout:4.18.0-372.9.1.el8.x86_64 2023-11-24T05:50:12.424 INFO:teuthology.task.kernel:Running kernel on smithi091: 4.18.0-372.9.1.el8.x86_64 2023-11-24T05:50:12.424 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum install -y kernel 2023-11-24T05:50:12.425 INFO:teuthology.orchestra.run.smithi094.stdout:4.18.0-372.9.1.el8.x86_64 2023-11-24T05:50:12.425 INFO:teuthology.task.kernel:Running kernel on smithi094: 4.18.0-372.9.1.el8.x86_64 2023-11-24T05:50:12.425 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum install -y kernel 2023-11-24T05:50:13.644 INFO:teuthology.orchestra.run.smithi091.stdout:Updating Subscription Management repositories. 2023-11-24T05:50:13.645 INFO:teuthology.orchestra.run.smithi091.stdout:Unable to read consumer identity 2023-11-24T05:50:13.664 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-11-24T05:50:13.664 INFO:teuthology.orchestra.run.smithi094.stdout:Unable to read consumer identity 2023-11-24T05:50:15.187 INFO:teuthology.orchestra.run.smithi094.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2023-11-24T05:50:15.200 INFO:teuthology.orchestra.run.smithi091.stdout:Copr repo for python3-asyncssh owned by ceph 11 kB/s | 3.5 kB 00:00 2023-11-24T05:50:15.887 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 23 MB/s | 16 MB 00:00 2023-11-24T05:50:16.063 INFO:teuthology.orchestra.run.smithi091.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2023-11-24T05:50:19.421 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 264 kB/s | 24 kB 00:00 2023-11-24T05:50:19.538 INFO:teuthology.orchestra.run.smithi091.stdout:lab-extras 560 kB/s | 24 kB 00:00 2023-11-24T05:50:20.156 INFO:teuthology.orchestra.run.smithi094.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-11-24T05:50:20.201 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-11-24T05:50:20.202 INFO:teuthology.orchestra.run.smithi094.stdout:Nothing to do. 2023-11-24T05:50:20.203 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-11-24T05:50:20.207 INFO:teuthology.orchestra.run.smithi091.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-11-24T05:50:20.260 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2023-11-24T05:50:20.261 INFO:teuthology.orchestra.run.smithi091.stdout:Nothing to do. 2023-11-24T05:50:20.261 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2023-11-24T05:50:20.325 DEBUG:teuthology.orchestra.run.smithi094:> echo no | sudo yum reinstall kernel || true 2023-11-24T05:50:20.378 DEBUG:teuthology.orchestra.run.smithi091:> echo no | sudo yum reinstall kernel || true 2023-11-24T05:50:20.653 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-11-24T05:50:20.653 INFO:teuthology.orchestra.run.smithi094.stdout:Unable to read consumer identity 2023-11-24T05:50:20.708 INFO:teuthology.orchestra.run.smithi091.stdout:Updating Subscription Management repositories. 2023-11-24T05:50:20.708 INFO:teuthology.orchestra.run.smithi091.stdout:Unable to read consumer identity 2023-11-24T05:50:20.909 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:01 ago on Fri 24 Nov 2023 05:50:19 AM UTC. 2023-11-24T05:50:20.957 INFO:teuthology.orchestra.run.smithi094.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-24T05:50:20.967 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:00:01 ago on Fri 24 Nov 2023 05:50:19 AM UTC. 2023-11-24T05:50:20.997 INFO:teuthology.orchestra.run.smithi094.stderr:Error: No packages marked for reinstall. 2023-11-24T05:50:21.009 INFO:teuthology.orchestra.run.smithi091.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-24T05:50:21.042 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum reinstall -y kernel || true 2023-11-24T05:50:21.068 INFO:teuthology.orchestra.run.smithi091.stderr:Error: No packages marked for reinstall. 2023-11-24T05:50:21.113 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum reinstall -y kernel || true 2023-11-24T05:50:21.373 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-11-24T05:50:21.373 INFO:teuthology.orchestra.run.smithi094.stdout:Unable to read consumer identity 2023-11-24T05:50:21.441 INFO:teuthology.orchestra.run.smithi091.stdout:Updating Subscription Management repositories. 2023-11-24T05:50:21.441 INFO:teuthology.orchestra.run.smithi091.stdout:Unable to read consumer identity 2023-11-24T05:50:21.630 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:00:02 ago on Fri 24 Nov 2023 05:50:19 AM UTC. 2023-11-24T05:50:21.678 INFO:teuthology.orchestra.run.smithi094.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-24T05:50:21.692 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:00:02 ago on Fri 24 Nov 2023 05:50:19 AM UTC. 2023-11-24T05:50:21.710 INFO:teuthology.orchestra.run.smithi094.stderr:Error: No packages marked for reinstall. 2023-11-24T05:50:21.732 INFO:teuthology.orchestra.run.smithi091.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-24T05:50:21.755 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q kernel | sort -rV | head -n 1 2023-11-24T05:50:21.759 INFO:teuthology.orchestra.run.smithi091.stderr:Error: No packages marked for reinstall. 2023-11-24T05:50:21.805 DEBUG:teuthology.orchestra.run.smithi091:> rpm -q kernel | sort -rV | head -n 1 2023-11-24T05:50:21.835 INFO:teuthology.orchestra.run.smithi094.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-11-24T05:50:21.835 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-11-24T05:50:21.835 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-11-24T05:50:21.835 DEBUG:teuthology.parallel:result is None 2023-11-24T05:50:21.886 INFO:teuthology.orchestra.run.smithi091.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-11-24T05:50:21.886 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-11-24T05:50:21.886 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-11-24T05:50:21.886 DEBUG:teuthology.parallel:result is None 2023-11-24T05:50:21.887 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-24T05:50:21.894 INFO:teuthology.task.internal:Creating test directory... 2023-11-24T05:50:21.894 DEBUG:teuthology.orchestra.run.smithi091:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-24T05:50:21.897 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-24T05:50:21.914 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-24T05:50:21.922 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-24T05:50:21.931 INFO:teuthology.task.internal:Creating archive directory... 2023-11-24T05:50:21.931 DEBUG:teuthology.orchestra.run.smithi091:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-24T05:50:21.954 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-24T05:50:21.977 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-24T05:50:21.985 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-24T05:50:21.985 DEBUG:teuthology.orchestra.run.smithi091:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-24T05:50:22.012 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-24T05:50:22.048 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-24T05:50:22.051 INFO:teuthology.orchestra.run.smithi091.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-24T05:50:22.062 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-24T05:50:22.066 INFO:teuthology.orchestra.run.smithi091.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-24T05:50:22.068 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-24T05:50:22.075 INFO:teuthology.task.internal:Configuring sudo... 2023-11-24T05:50:22.075 DEBUG:teuthology.orchestra.run.smithi091:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-24T05:50:22.115 DEBUG:teuthology.orchestra.run.smithi094:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-24T05:50:22.142 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-24T05:50:22.156 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-24T05:50:22.157 DEBUG:teuthology.orchestra.run.smithi091:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-24T05:50:22.181 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-24T05:50:22.199 DEBUG:teuthology.orchestra.run.smithi091:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-24T05:50:22.281 DEBUG:teuthology.orchestra.run.smithi091:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-24T05:50:22.327 DEBUG:teuthology.orchestra.run.smithi091:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-24T05:50:22.408 DEBUG:teuthology.orchestra.run.smithi091:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-24T05:50:22.454 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T05:50:22.454 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-24T05:50:22.522 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-24T05:50:22.571 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-24T05:50:22.614 DEBUG:teuthology.orchestra.run.smithi094:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-24T05:50:22.696 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-24T05:50:22.740 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T05:50:22.740 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-24T05:50:22.808 DEBUG:teuthology.orchestra.run.smithi091:> sudo service rsyslog restart 2023-11-24T05:50:22.810 DEBUG:teuthology.orchestra.run.smithi094:> sudo service rsyslog restart 2023-11-24T05:50:22.860 INFO:teuthology.orchestra.run.smithi091.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-24T05:50:22.902 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-24T05:50:23.263 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-24T05:50:23.271 INFO:teuthology.task.internal:Starting timer... 2023-11-24T05:50:23.271 INFO:teuthology.run_tasks:Running task pcp... 2023-11-24T05:50:23.285 INFO:teuthology.run_tasks:Running task selinux... 2023-11-24T05:50:23.295 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-24T05:50:23.297 DEBUG:teuthology.orchestra.run.smithi091:> sudo service auditd rotate 2023-11-24T05:50:23.348 INFO:teuthology.orchestra.run.smithi091.stdout:Rotating logs: 2023-11-24T05:50:23.350 DEBUG:teuthology.orchestra.run.smithi094:> sudo service auditd rotate 2023-11-24T05:50:23.405 INFO:teuthology.orchestra.run.smithi094.stdout:Rotating logs: 2023-11-24T05:50:23.407 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-24T05:50:23.407 DEBUG:teuthology.orchestra.run.smithi091:> /usr/sbin/getenforce 2023-11-24T05:50:23.430 INFO:teuthology.orchestra.run.smithi091.stdout:Permissive 2023-11-24T05:50:23.431 DEBUG:teuthology.orchestra.run.smithi094:> /usr/sbin/getenforce 2023-11-24T05:50:23.455 INFO:teuthology.orchestra.run.smithi094.stdout:Permissive 2023-11-24T05:50:23.456 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi091.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi094.front.sepia.ceph.com': 'permissive'} 2023-11-24T05:50:23.456 DEBUG:teuthology.orchestra.run.smithi091:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-24T05:50:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T05:50:23.501 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-24T05:50:23.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T05:50:23.528 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-24T05:50:23.528 DEBUG:teuthology.orchestra.run.smithi091:> sudo /usr/sbin/setenforce permissive 2023-11-24T05:50:23.573 DEBUG:teuthology.orchestra.run.smithi094:> sudo /usr/sbin/setenforce permissive 2023-11-24T05:50:23.599 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-24T05:50:23.613 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-11-24T05:50:23.652 INFO:teuthology.repo_utils:Fetching main from origin 2023-11-24T05:50:24.089 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-24T05:50:24.159 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-24T05:50:24.160 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi091.front.sepia.ceph.com,smithi094.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-24T06:04:24.836 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi091.front.sepia.ceph.com'), Remote(name='ubuntu@smithi094.front.sepia.ceph.com')] 2023-11-24T06:04:24.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi091.front.sepia.ceph.com' 2023-11-24T06:04:24.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi091.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T06:04:24.907 DEBUG:teuthology.orchestra.run.smithi091:> true 2023-11-24T06:04:24.931 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi091.front.sepia.ceph.com' 2023-11-24T06:04:24.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2023-11-24T06:04:24.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-24T06:04:24.999 DEBUG:teuthology.orchestra.run.smithi094:> true 2023-11-24T06:04:25.072 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi094.front.sepia.ceph.com' 2023-11-24T06:04:25.072 INFO:teuthology.run_tasks:Running task clock... 2023-11-24T06:04:25.084 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-24T06:04:25.084 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-24T06:04:25.085 DEBUG:teuthology.orchestra.run.smithi091:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-24T06:04:25.087 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-24T06:04:25.087 DEBUG:teuthology.orchestra.run.smithi094:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-24T06:04:25.117 INFO:teuthology.orchestra.run.smithi091.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-24T06:04:25.131 INFO:teuthology.orchestra.run.smithi091.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-24T06:04:25.143 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-24T06:04:25.157 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-24T06:04:25.205 INFO:teuthology.orchestra.run.smithi091.stderr:sudo: ntpd: command not found 2023-11-24T06:04:25.216 INFO:teuthology.orchestra.run.smithi091.stdout:506 Cannot talk to daemon 2023-11-24T06:04:25.226 INFO:teuthology.orchestra.run.smithi094.stderr:sudo: ntpd: command not found 2023-11-24T06:04:25.230 INFO:teuthology.orchestra.run.smithi091.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-24T06:04:25.238 INFO:teuthology.orchestra.run.smithi094.stdout:506 Cannot talk to daemon 2023-11-24T06:04:25.244 INFO:teuthology.orchestra.run.smithi091.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-24T06:04:25.253 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-24T06:04:25.267 INFO:teuthology.orchestra.run.smithi094.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-24T06:04:25.328 INFO:teuthology.orchestra.run.smithi091.stderr:bash: ntpq: command not found 2023-11-24T06:04:25.331 INFO:teuthology.orchestra.run.smithi091.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-24T06:04:25.331 INFO:teuthology.orchestra.run.smithi091.stdout:=============================================================================== 2023-11-24T06:04:25.331 INFO:teuthology.orchestra.run.smithi091.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:04:25.331 INFO:teuthology.orchestra.run.smithi091.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:04:25.331 INFO:teuthology.orchestra.run.smithi091.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:04:25.331 INFO:teuthology.orchestra.run.smithi091.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:04:25.344 INFO:teuthology.orchestra.run.smithi094.stderr:bash: ntpq: command not found 2023-11-24T06:04:25.348 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-24T06:04:25.348 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2023-11-24T06:04:25.348 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:04:25.348 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:04:25.349 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:04:25.349 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:04:25.350 INFO:teuthology.run_tasks:Running task install... 2023-11-24T06:04:25.358 DEBUG:teuthology.task.install:project ceph 2023-11-24T06:04:25.361 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'}} 2023-11-24T06:04:25.361 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'} 2023-11-24T06:04:25.361 INFO:teuthology.task.install:Using flavor: default 2023-11-24T06:04:25.367 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-24T06:04:25.368 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-11-24T06:04:25.368 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c', 'tag': None, 'wait_for_package': False} 2023-11-24T06:04:25.369 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a13ea5ff00cb4df7a9cc941166ddeb358881144c 2023-11-24T06:04:25.370 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c', 'tag': None, 'wait_for_package': False} 2023-11-24T06:04:25.370 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a13ea5ff00cb4df7a9cc941166ddeb358881144c 2023-11-24T06:04:25.526 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/8/flavors/default/ 2023-11-24T06:04:25.527 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7484.ga13ea5ff 2023-11-24T06:04:25.646 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/8/flavors/default/ 2023-11-24T06:04:25.647 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7484.ga13ea5ff 2023-11-24T06:04:25.753 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-24T06:04:25.754 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:04:25.754 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-24T06:04:25.782 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-24T06:04:25.783 DEBUG:teuthology.orchestra.run.smithi094:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a13ea5ff00cb4df7a9cc941166ddeb358881144c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-24T06:04:25.851 DEBUG:teuthology.orchestra.run.smithi094:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-24T06:04:25.871 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-mgr-vol-misc/a13ea5ff00cb4df7a9cc941166ddeb358881144c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-24T06:04:25.871 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:04:25.871 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-24T06:04:25.901 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-11-24T06:04:25.901 DEBUG:teuthology.orchestra.run.smithi091:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a13ea5ff00cb4df7a9cc941166ddeb358881144c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-24T06:04:25.928 DEBUG:teuthology.orchestra.run.smithi094:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-24T06:04:25.971 DEBUG:teuthology.orchestra.run.smithi091:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-24T06:04:25.994 INFO:teuthology.orchestra.run.smithi094.stdout:check_obsoletes = 1 2023-11-24T06:04:25.995 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum clean all 2023-11-24T06:04:26.050 DEBUG:teuthology.orchestra.run.smithi091:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-24T06:04:26.117 INFO:teuthology.orchestra.run.smithi091.stdout:check_obsoletes = 1 2023-11-24T06:04:26.118 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum clean all 2023-11-24T06:04:26.367 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-11-24T06:04:26.487 INFO:teuthology.orchestra.run.smithi091.stdout:Updating Subscription Management repositories. 2023-11-24T06:04:27.054 INFO:teuthology.orchestra.run.smithi094.stdout:44 files removed 2023-11-24T06:04:27.102 DEBUG:teuthology.orchestra.run.smithi094:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-24T06:04:27.349 INFO:teuthology.orchestra.run.smithi091.stdout:44 files removed 2023-11-24T06:04:27.399 DEBUG:teuthology.orchestra.run.smithi091:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-11-24T06:04:27.432 INFO:teuthology.orchestra.run.smithi094.stdout:Updating Subscription Management repositories. 2023-11-24T06:04:27.730 INFO:teuthology.orchestra.run.smithi091.stdout:Updating Subscription Management repositories. 2023-11-24T06:04:28.737 INFO:teuthology.orchestra.run.smithi091.stdout:ceph packages for x86_64 286 kB/s | 81 kB 00:00 2023-11-24T06:04:28.840 INFO:teuthology.orchestra.run.smithi094.stdout:ceph packages for x86_64 169 kB/s | 81 kB 00:00 2023-11-24T06:04:29.163 INFO:teuthology.orchestra.run.smithi091.stdout:ceph noarch packages 24 kB/s | 9.5 kB 00:00 2023-11-24T06:04:29.215 INFO:teuthology.orchestra.run.smithi094.stdout:ceph noarch packages 28 kB/s | 9.5 kB 00:00 2023-11-24T06:04:29.475 INFO:teuthology.orchestra.run.smithi091.stdout:ceph source packages 5.1 kB/s | 1.5 kB 00:00 2023-11-24T06:04:29.533 INFO:teuthology.orchestra.run.smithi094.stdout:ceph source packages 5.0 kB/s | 1.5 kB 00:00 2023-11-24T06:04:29.723 INFO:teuthology.orchestra.run.smithi091.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2023-11-24T06:04:29.776 INFO:teuthology.orchestra.run.smithi094.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2023-11-24T06:04:33.376 INFO:teuthology.orchestra.run.smithi091.stdout:Extra Packages for Enterprise Linux 4.4 MB/s | 16 MB 00:03 2023-11-24T06:04:36.915 INFO:teuthology.orchestra.run.smithi091.stdout:lab-extras 388 kB/s | 24 kB 00:00 2023-11-24T06:04:37.850 INFO:teuthology.orchestra.run.smithi091.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 MB/s | 53 MB 00:00 2023-11-24T06:04:45.884 INFO:teuthology.orchestra.run.smithi091.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 61 MB/s | 47 MB 00:00 2023-11-24T06:04:49.028 INFO:teuthology.orchestra.run.smithi094.stdout:Extra Packages for Enterprise Linux 849 kB/s | 16 MB 00:19 2023-11-24T06:04:52.584 INFO:teuthology.orchestra.run.smithi094.stdout:lab-extras 385 kB/s | 24 kB 00:00 2023-11-24T06:04:53.437 INFO:teuthology.orchestra.run.smithi094.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 61 MB/s | 47 MB 00:00 2023-11-24T06:04:56.752 INFO:teuthology.orchestra.run.smithi091.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-24T06:04:56.752 INFO:teuthology.orchestra.run.smithi091.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-24T06:04:56.767 INFO:teuthology.orchestra.run.smithi091.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-24T06:04:56.768 INFO:teuthology.orchestra.run.smithi091.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-24T06:04:56.769 INFO:teuthology.orchestra.run.smithi091.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-24T06:04:56.770 INFO:teuthology.orchestra.run.smithi091.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-24T06:04:56.772 INFO:teuthology.orchestra.run.smithi091.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-24T06:04:56.775 INFO:teuthology.orchestra.run.smithi091.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-24T06:04:56.776 INFO:teuthology.orchestra.run.smithi091.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-24T06:04:56.907 INFO:teuthology.orchestra.run.smithi091.stdout:Dependencies resolved. 2023-11-24T06:04:56.920 INFO:teuthology.orchestra.run.smithi091.stdout:=========================================================================================================================== 2023-11-24T06:04:56.920 INFO:teuthology.orchestra.run.smithi091.stdout: Package Arch Version Repository Size 2023-11-24T06:04:56.920 INFO:teuthology.orchestra.run.smithi091.stdout:=========================================================================================================================== 2023-11-24T06:04:56.920 INFO:teuthology.orchestra.run.smithi091.stdout:Installing: 2023-11-24T06:04:56.920 INFO:teuthology.orchestra.run.smithi091.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-24T06:04:56.920 INFO:teuthology.orchestra.run.smithi091.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-24T06:04:56.920 INFO:teuthology.orchestra.run.smithi091.stdout: ceph x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 6.4 k 2023-11-24T06:04:56.921 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-base x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 5.4 M 2023-11-24T06:04:56.921 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-fuse x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 920 k 2023-11-24T06:04:56.921 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 171 k 2023-11-24T06:04:56.921 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 1.6 M 2023-11-24T06:04:56.921 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 133 k 2023-11-24T06:04:56.921 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 1.8 M 2023-11-24T06:04:56.921 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 7.4 M 2023-11-24T06:04:56.921 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-rook noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 50 k 2023-11-24T06:04:56.921 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-radosgw x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 14 M 2023-11-24T06:04:56.921 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-test x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 49 M 2023-11-24T06:04:56.921 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-volume noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 262 k 2023-11-24T06:04:56.921 INFO:teuthology.orchestra.run.smithi091.stdout: cephadm noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 407 k 2023-11-24T06:04:56.922 INFO:teuthology.orchestra.run.smithi091.stdout: cephfs-mirror x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 262 k 2023-11-24T06:04:56.922 INFO:teuthology.orchestra.run.smithi091.stdout: cephfs-top noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 18 k 2023-11-24T06:04:56.922 INFO:teuthology.orchestra.run.smithi091.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-24T06:04:56.922 INFO:teuthology.orchestra.run.smithi091.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-24T06:04:56.922 INFO:teuthology.orchestra.run.smithi091.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-24T06:04:56.922 INFO:teuthology.orchestra.run.smithi091.stdout: libcephfs-devel x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 31 k 2023-11-24T06:04:56.922 INFO:teuthology.orchestra.run.smithi091.stdout: libcephfs2 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 839 k 2023-11-24T06:04:56.922 INFO:teuthology.orchestra.run.smithi091.stdout: librados-devel x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 139 k 2023-11-24T06:04:56.922 INFO:teuthology.orchestra.run.smithi091.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-24T06:04:56.922 INFO:teuthology.orchestra.run.smithi091.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-24T06:04:56.922 INFO:teuthology.orchestra.run.smithi091.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-24T06:04:56.923 INFO:teuthology.orchestra.run.smithi091.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-24T06:04:56.923 INFO:teuthology.orchestra.run.smithi091.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-24T06:04:56.923 INFO:teuthology.orchestra.run.smithi091.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-24T06:04:56.923 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cephfs x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 197 k 2023-11-24T06:04:56.923 INFO:teuthology.orchestra.run.smithi091.stdout: python3-rados x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 371 k 2023-11-24T06:04:56.923 INFO:teuthology.orchestra.run.smithi091.stdout: python3-rbd x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 364 k 2023-11-24T06:04:56.923 INFO:teuthology.orchestra.run.smithi091.stdout: python3-rgw x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 107 k 2023-11-24T06:04:56.923 INFO:teuthology.orchestra.run.smithi091.stdout: rbd-fuse x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 94 k 2023-11-24T06:04:56.923 INFO:teuthology.orchestra.run.smithi091.stdout: rbd-mirror x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 3.6 M 2023-11-24T06:04:56.923 INFO:teuthology.orchestra.run.smithi091.stdout: rbd-nbd x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 181 k 2023-11-24T06:04:56.923 INFO:teuthology.orchestra.run.smithi091.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-24T06:04:56.924 INFO:teuthology.orchestra.run.smithi091.stdout:Upgrading: 2023-11-24T06:04:56.924 INFO:teuthology.orchestra.run.smithi091.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-24T06:04:56.924 INFO:teuthology.orchestra.run.smithi091.stdout: librados2 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 3.8 M 2023-11-24T06:04:56.924 INFO:teuthology.orchestra.run.smithi091.stdout: librbd1 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 3.7 M 2023-11-24T06:04:56.924 INFO:teuthology.orchestra.run.smithi091.stdout:Installing dependencies: 2023-11-24T06:04:56.924 INFO:teuthology.orchestra.run.smithi091.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-24T06:04:56.924 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-common x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 23 M 2023-11-24T06:04:56.924 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 28 k 2023-11-24T06:04:56.924 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mds x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 2.3 M 2023-11-24T06:04:56.924 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 243 k 2023-11-24T06:04:56.925 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mon x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 4.6 M 2023-11-24T06:04:56.925 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-osd x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 18 M 2023-11-24T06:04:56.925 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 15 k 2023-11-24T06:04:56.925 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-selinux x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 24 k 2023-11-24T06:04:56.925 INFO:teuthology.orchestra.run.smithi091.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-24T06:04:56.925 INFO:teuthology.orchestra.run.smithi091.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-24T06:04:56.925 INFO:teuthology.orchestra.run.smithi091.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-24T06:04:56.925 INFO:teuthology.orchestra.run.smithi091.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-24T06:04:56.925 INFO:teuthology.orchestra.run.smithi091.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-24T06:04:56.925 INFO:teuthology.orchestra.run.smithi091.stdout: libcephsqlite x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 183 k 2023-11-24T06:04:56.925 INFO:teuthology.orchestra.run.smithi091.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-24T06:04:56.926 INFO:teuthology.orchestra.run.smithi091.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-24T06:04:56.926 INFO:teuthology.orchestra.run.smithi091.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-24T06:04:56.926 INFO:teuthology.orchestra.run.smithi091.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-24T06:04:56.926 INFO:teuthology.orchestra.run.smithi091.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-24T06:04:56.926 INFO:teuthology.orchestra.run.smithi091.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-24T06:04:56.926 INFO:teuthology.orchestra.run.smithi091.stdout: libradosstriper1 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 517 k 2023-11-24T06:04:56.926 INFO:teuthology.orchestra.run.smithi091.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-24T06:04:56.926 INFO:teuthology.orchestra.run.smithi091.stdout: librgw2 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 6.9 M 2023-11-24T06:04:56.926 INFO:teuthology.orchestra.run.smithi091.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-24T06:04:56.926 INFO:teuthology.orchestra.run.smithi091.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-24T06:04:56.927 INFO:teuthology.orchestra.run.smithi091.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-24T06:04:56.927 INFO:teuthology.orchestra.run.smithi091.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-24T06:04:56.927 INFO:teuthology.orchestra.run.smithi091.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-24T06:04:56.927 INFO:teuthology.orchestra.run.smithi091.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-24T06:04:56.927 INFO:teuthology.orchestra.run.smithi091.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-24T06:04:56.927 INFO:teuthology.orchestra.run.smithi091.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-24T06:04:56.927 INFO:teuthology.orchestra.run.smithi091.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-24T06:04:56.927 INFO:teuthology.orchestra.run.smithi091.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-24T06:04:56.927 INFO:teuthology.orchestra.run.smithi091.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-24T06:04:56.927 INFO:teuthology.orchestra.run.smithi091.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-24T06:04:56.927 INFO:teuthology.orchestra.run.smithi091.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-24T06:04:56.927 INFO:teuthology.orchestra.run.smithi091.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-24T06:04:56.928 INFO:teuthology.orchestra.run.smithi091.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-24T06:04:56.928 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-24T06:04:56.928 INFO:teuthology.orchestra.run.smithi091.stdout: python3-ceph-argparse x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 45 k 2023-11-24T06:04:56.928 INFO:teuthology.orchestra.run.smithi091.stdout: python3-ceph-common x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 128 k 2023-11-24T06:04:56.928 INFO:teuthology.orchestra.run.smithi091.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-24T06:04:56.928 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-24T06:04:56.928 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-24T06:04:56.928 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-24T06:04:56.928 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-24T06:04:56.928 INFO:teuthology.orchestra.run.smithi091.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-24T06:04:56.928 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-24T06:04:56.929 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-24T06:04:56.929 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-24T06:04:56.929 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-24T06:04:56.929 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-24T06:04:56.929 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-24T06:04:56.929 INFO:teuthology.orchestra.run.smithi091.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-24T06:04:56.929 INFO:teuthology.orchestra.run.smithi091.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-24T06:04:56.929 INFO:teuthology.orchestra.run.smithi091.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-24T06:04:56.929 INFO:teuthology.orchestra.run.smithi091.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-24T06:04:56.929 INFO:teuthology.orchestra.run.smithi091.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-24T06:04:56.929 INFO:teuthology.orchestra.run.smithi091.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-24T06:04:56.930 INFO:teuthology.orchestra.run.smithi091.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-24T06:04:56.930 INFO:teuthology.orchestra.run.smithi091.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-24T06:04:56.930 INFO:teuthology.orchestra.run.smithi091.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-24T06:04:56.930 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-24T06:04:56.930 INFO:teuthology.orchestra.run.smithi091.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-24T06:04:56.930 INFO:teuthology.orchestra.run.smithi091.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-24T06:04:56.930 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-24T06:04:56.930 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-24T06:04:56.930 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-24T06:04:56.930 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-24T06:04:56.930 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-24T06:04:56.931 INFO:teuthology.orchestra.run.smithi091.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-24T06:04:56.931 INFO:teuthology.orchestra.run.smithi091.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-24T06:04:56.931 INFO:teuthology.orchestra.run.smithi091.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-24T06:04:56.931 INFO:teuthology.orchestra.run.smithi091.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-24T06:04:56.931 INFO:teuthology.orchestra.run.smithi091.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-24T06:04:56.931 INFO:teuthology.orchestra.run.smithi091.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-24T06:04:56.931 INFO:teuthology.orchestra.run.smithi091.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-24T06:04:56.931 INFO:teuthology.orchestra.run.smithi091.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-24T06:04:56.931 INFO:teuthology.orchestra.run.smithi091.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-24T06:04:56.931 INFO:teuthology.orchestra.run.smithi091.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-24T06:04:56.931 INFO:teuthology.orchestra.run.smithi091.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-24T06:04:56.932 INFO:teuthology.orchestra.run.smithi091.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-24T06:04:56.932 INFO:teuthology.orchestra.run.smithi091.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-24T06:04:56.932 INFO:teuthology.orchestra.run.smithi091.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-24T06:04:56.932 INFO:teuthology.orchestra.run.smithi091.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-24T06:04:56.932 INFO:teuthology.orchestra.run.smithi091.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-24T06:04:56.932 INFO:teuthology.orchestra.run.smithi091.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-24T06:04:56.932 INFO:teuthology.orchestra.run.smithi091.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-24T06:04:56.932 INFO:teuthology.orchestra.run.smithi091.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-24T06:04:56.932 INFO:teuthology.orchestra.run.smithi091.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-24T06:04:56.932 INFO:teuthology.orchestra.run.smithi091.stdout:Enabling module streams: 2023-11-24T06:04:56.932 INFO:teuthology.orchestra.run.smithi091.stdout: postgresql 10 2023-11-24T06:04:56.932 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:04:56.933 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction Summary 2023-11-24T06:04:56.933 INFO:teuthology.orchestra.run.smithi091.stdout:=========================================================================================================================== 2023-11-24T06:04:56.933 INFO:teuthology.orchestra.run.smithi091.stdout:Install 128 Packages 2023-11-24T06:04:56.933 INFO:teuthology.orchestra.run.smithi091.stdout:Upgrade 3 Packages 2023-11-24T06:04:56.933 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:04:56.933 INFO:teuthology.orchestra.run.smithi091.stdout:Total download size: 209 M 2023-11-24T06:04:56.933 INFO:teuthology.orchestra.run.smithi091.stdout:Downloading Packages: 2023-11-24T06:04:57.241 INFO:teuthology.orchestra.run.smithi091.stdout:(1/131): ceph-18.0.0-7484.ga13ea5ff.el8.x86_64. 23 kB/s | 6.4 kB 00:00 2023-11-24T06:04:57.783 INFO:teuthology.orchestra.run.smithi091.stdout:(2/131): ceph-fuse-18.0.0-7484.ga13ea5ff.el8.x8 1.7 MB/s | 920 kB 00:00 2023-11-24T06:04:57.942 INFO:teuthology.orchestra.run.smithi091.stdout:(3/131): ceph-immutable-object-cache-18.0.0-748 1.1 MB/s | 171 kB 00:00 2023-11-24T06:04:58.733 INFO:teuthology.orchestra.run.smithi091.stdout:(4/131): ceph-common-18.0.0-7484.ga13ea5ff.el8. 13 MB/s | 23 MB 00:01 2023-11-24T06:04:58.847 INFO:teuthology.orchestra.run.smithi091.stdout:(5/131): ceph-mds-18.0.0-7484.ga13ea5ff.el8.x86 2.6 MB/s | 2.3 MB 00:00 2023-11-24T06:04:58.926 INFO:teuthology.orchestra.run.smithi091.stdout:(6/131): ceph-base-18.0.0-7484.ga13ea5ff.el8.x8 2.7 MB/s | 5.4 MB 00:01 2023-11-24T06:04:58.968 INFO:teuthology.orchestra.run.smithi091.stdout:(7/131): ceph-mgr-18.0.0-7484.ga13ea5ff.el8.x86 7.0 MB/s | 1.6 MB 00:00 2023-11-24T06:04:59.851 INFO:teuthology.orchestra.run.smithi091.stdout:(8/131): ceph-radosgw-18.0.0-7484.ga13ea5ff.el8 16 MB/s | 14 MB 00:00 2023-11-24T06:04:59.918 INFO:teuthology.orchestra.run.smithi091.stdout:(9/131): ceph-mon-18.0.0-7484.ga13ea5ff.el8.x86 4.3 MB/s | 4.6 MB 00:01 2023-11-24T06:04:59.943 INFO:teuthology.orchestra.run.smithi091.stdout:(10/131): ceph-selinux-18.0.0-7484.ga13ea5ff.el 265 kB/s | 24 kB 00:00 2023-11-24T06:05:00.086 INFO:teuthology.orchestra.run.smithi091.stdout:(11/131): cephfs-mirror-18.0.0-7484.ga13ea5ff.e 1.8 MB/s | 262 kB 00:00 2023-11-24T06:05:00.136 INFO:teuthology.orchestra.run.smithi091.stdout:(12/131): libcephfs-devel-18.0.0-7484.ga13ea5ff 622 kB/s | 31 kB 00:00 2023-11-24T06:05:00.244 INFO:teuthology.orchestra.run.smithi091.stdout:(13/131): libcephfs2-18.0.0-7484.ga13ea5ff.el8. 7.6 MB/s | 839 kB 00:00 2023-11-24T06:05:00.311 INFO:teuthology.orchestra.run.smithi091.stdout:(14/131): libcephsqlite-18.0.0-7484.ga13ea5ff.e 2.7 MB/s | 183 kB 00:00 2023-11-24T06:05:00.370 INFO:teuthology.orchestra.run.smithi091.stdout:(15/131): librados-devel-18.0.0-7484.ga13ea5ff. 2.3 MB/s | 139 kB 00:00 2023-11-24T06:05:00.429 INFO:teuthology.orchestra.run.smithi091.stdout:(16/131): libradosstriper1-18.0.0-7484.ga13ea5f 8.7 MB/s | 517 kB 00:00 2023-11-24T06:05:00.862 INFO:teuthology.orchestra.run.smithi091.stdout:(17/131): librgw2-18.0.0-7484.ga13ea5ff.el8.x86 16 MB/s | 6.9 MB 00:00 2023-11-24T06:05:00.913 INFO:teuthology.orchestra.run.smithi091.stdout:(18/131): python3-ceph-argparse-18.0.0-7484.ga1 900 kB/s | 45 kB 00:00 2023-11-24T06:05:00.963 INFO:teuthology.orchestra.run.smithi091.stdout:(19/131): python3-ceph-common-18.0.0-7484.ga13e 2.5 MB/s | 128 kB 00:00 2023-11-24T06:05:01.013 INFO:teuthology.orchestra.run.smithi091.stdout:(20/131): python3-cephfs-18.0.0-7484.ga13ea5ff. 3.9 MB/s | 197 kB 00:00 2023-11-24T06:05:01.072 INFO:teuthology.orchestra.run.smithi091.stdout:(21/131): python3-rados-18.0.0-7484.ga13ea5ff.e 6.2 MB/s | 371 kB 00:00 2023-11-24T06:05:01.172 INFO:teuthology.orchestra.run.smithi091.stdout:(22/131): python3-rbd-18.0.0-7484.ga13ea5ff.el8 3.6 MB/s | 364 kB 00:00 2023-11-24T06:05:01.239 INFO:teuthology.orchestra.run.smithi091.stdout:(23/131): python3-rgw-18.0.0-7484.ga13ea5ff.el8 1.6 MB/s | 107 kB 00:00 2023-11-24T06:05:01.306 INFO:teuthology.orchestra.run.smithi091.stdout:(24/131): rbd-fuse-18.0.0-7484.ga13ea5ff.el8.x8 1.4 MB/s | 94 kB 00:00 2023-11-24T06:05:01.347 INFO:teuthology.orchestra.run.smithi094.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 MB/s | 53 MB 00:00 2023-11-24T06:05:01.573 INFO:teuthology.orchestra.run.smithi091.stdout:(25/131): rbd-mirror-18.0.0-7484.ga13ea5ff.el8. 13 MB/s | 3.6 MB 00:00 2023-11-24T06:05:01.656 INFO:teuthology.orchestra.run.smithi091.stdout:(26/131): rbd-nbd-18.0.0-7484.ga13ea5ff.el8.x86 2.1 MB/s | 181 kB 00:00 2023-11-24T06:05:01.849 INFO:teuthology.orchestra.run.smithi091.stdout:(27/131): ceph-osd-18.0.0-7484.ga13ea5ff.el8.x8 6.2 MB/s | 18 MB 00:02 2023-11-24T06:05:01.873 INFO:teuthology.orchestra.run.smithi091.stdout:(28/131): ceph-grafana-dashboards-18.0.0-7484.g 129 kB/s | 28 kB 00:00 2023-11-24T06:05:01.966 INFO:teuthology.orchestra.run.smithi091.stdout:(29/131): ceph-mgr-cephadm-18.0.0-7484.ga13ea5f 1.1 MB/s | 133 kB 00:00 2023-11-24T06:05:02.132 INFO:teuthology.orchestra.run.smithi091.stdout:(30/131): ceph-mgr-dashboard-18.0.0-7484.ga13ea 6.8 MB/s | 1.8 MB 00:00 2023-11-24T06:05:02.241 INFO:teuthology.orchestra.run.smithi091.stdout:(31/131): ceph-mgr-modules-core-18.0.0-7484.ga1 2.2 MB/s | 243 kB 00:00 2023-11-24T06:05:02.325 INFO:teuthology.orchestra.run.smithi091.stdout:(32/131): ceph-mgr-rook-18.0.0-7484.ga13ea5ff.e 601 kB/s | 50 kB 00:00 2023-11-24T06:05:02.392 INFO:teuthology.orchestra.run.smithi091.stdout:(33/131): ceph-prometheus-alerts-18.0.0-7484.ga 225 kB/s | 15 kB 00:00 2023-11-24T06:05:02.484 INFO:teuthology.orchestra.run.smithi091.stdout:(34/131): ceph-volume-18.0.0-7484.ga13ea5ff.el8 2.8 MB/s | 262 kB 00:00 2023-11-24T06:05:02.584 INFO:teuthology.orchestra.run.smithi091.stdout:(35/131): cephadm-18.0.0-7484.ga13ea5ff.el8.noa 4.0 MB/s | 407 kB 00:00 2023-11-24T06:05:02.626 INFO:teuthology.orchestra.run.smithi091.stdout:(36/131): cephfs-top-18.0.0-7484.ga13ea5ff.el8. 437 kB/s | 18 kB 00:00 2023-11-24T06:05:02.843 INFO:teuthology.orchestra.run.smithi091.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2023-11-24T06:05:02.901 INFO:teuthology.orchestra.run.smithi091.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2023-11-24T06:05:02.960 INFO:teuthology.orchestra.run.smithi091.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2023-11-24T06:05:03.069 INFO:teuthology.orchestra.run.smithi091.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 4.3 MB/s | 476 kB 00:00 2023-11-24T06:05:03.144 INFO:teuthology.orchestra.run.smithi091.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 583 kB/s | 44 kB 00:00 2023-11-24T06:05:03.194 INFO:teuthology.orchestra.run.smithi091.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 3.6 MB/s | 185 kB 00:00 2023-11-24T06:05:03.319 INFO:teuthology.orchestra.run.smithi091.stdout:(43/131): ceph-mgr-diskprediction-local-18.0.0- 5.5 MB/s | 7.4 MB 00:01 2023-11-24T06:05:03.345 INFO:teuthology.orchestra.run.smithi091.stdout:(44/131): python3-cachetools-3.1.1-4.el8.noarch 219 kB/s | 33 kB 00:00 2023-11-24T06:05:03.429 INFO:teuthology.orchestra.run.smithi091.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.1 MB/s | 173 kB 00:00 2023-11-24T06:05:03.470 INFO:teuthology.orchestra.run.smithi091.stdout:(46/131): python3-certifi-2018.10.15-7.el8.noar 108 kB/s | 16 kB 00:00 2023-11-24T06:05:03.496 INFO:teuthology.orchestra.run.smithi091.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 5.6 MB/s | 384 kB 00:00 2023-11-24T06:05:03.546 INFO:teuthology.orchestra.run.smithi091.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 223 kB/s | 11 kB 00:00 2023-11-24T06:05:03.571 INFO:teuthology.orchestra.run.smithi091.stdout:(49/131): python3-google-auth-1.1.1-10.el8.noar 808 kB/s | 81 kB 00:00 2023-11-24T06:05:03.622 INFO:teuthology.orchestra.run.smithi091.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 234 kB/s | 18 kB 00:00 2023-11-24T06:05:03.680 INFO:teuthology.orchestra.run.smithi091.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 837 kB/s | 49 kB 00:00 2023-11-24T06:05:03.730 INFO:teuthology.orchestra.run.smithi091.stdout:(52/131): python3-kubernetes-11.0.0-6.el8.noarc 7.3 MB/s | 1.2 MB 00:00 2023-11-24T06:05:03.781 INFO:teuthology.orchestra.run.smithi091.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 586 kB/s | 59 kB 00:00 2023-11-24T06:05:03.806 INFO:teuthology.orchestra.run.smithi091.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 800 kB/s | 60 kB 00:00 2023-11-24T06:05:03.857 INFO:teuthology.orchestra.run.smithi091.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 3.7 MB/s | 283 kB 00:00 2023-11-24T06:05:03.882 INFO:teuthology.orchestra.run.smithi091.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 207 kB/s | 16 kB 00:00 2023-11-24T06:05:03.957 INFO:teuthology.orchestra.run.smithi091.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 336 kB/s | 34 kB 00:00 2023-11-24T06:05:03.982 INFO:teuthology.orchestra.run.smithi091.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 1.9 MB/s | 196 kB 00:00 2023-11-24T06:05:04.041 INFO:teuthology.orchestra.run.smithi091.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 771 kB/s | 64 kB 00:00 2023-11-24T06:05:04.066 INFO:teuthology.orchestra.run.smithi091.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 224 kB/s | 19 kB 00:00 2023-11-24T06:05:04.091 INFO:teuthology.orchestra.run.smithi091.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 475 kB/s | 24 kB 00:00 2023-11-24T06:05:04.142 INFO:teuthology.orchestra.run.smithi091.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 387 kB/s | 29 kB 00:00 2023-11-24T06:05:04.167 INFO:teuthology.orchestra.run.smithi091.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 364 kB/s | 27 kB 00:00 2023-11-24T06:05:04.209 INFO:teuthology.orchestra.run.smithi091.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 3.7 MB/s | 255 kB 00:00 2023-11-24T06:05:04.251 INFO:teuthology.orchestra.run.smithi091.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.9 MB/s | 251 kB 00:00 2023-11-24T06:05:04.276 INFO:teuthology.orchestra.run.smithi091.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 916 kB/s | 61 kB 00:00 2023-11-24T06:05:04.302 INFO:teuthology.orchestra.run.smithi091.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2023-11-24T06:05:04.344 INFO:teuthology.orchestra.run.smithi091.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 343 kB/s | 23 kB 00:00 2023-11-24T06:05:04.436 INFO:teuthology.orchestra.run.smithi091.stdout:(69/131): thrift-0.13.0-2.el8.x86_64.rpm 13 MB/s | 1.7 MB 00:00 2023-11-24T06:05:04.461 INFO:teuthology.orchestra.run.smithi091.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 595 kB/s | 70 kB 00:00 2023-11-24T06:05:04.520 INFO:teuthology.orchestra.run.smithi091.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 561 kB/s | 47 kB 00:00 2023-11-24T06:05:04.536 INFO:teuthology.orchestra.run.smithi091.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 2.6 MB/s | 200 kB 00:00 2023-11-24T06:05:04.570 INFO:teuthology.orchestra.run.smithi091.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 847 kB/s | 43 kB 00:00 2023-11-24T06:05:04.629 INFO:teuthology.orchestra.run.smithi091.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.3 MB/s | 220 kB 00:00 2023-11-24T06:05:04.654 INFO:teuthology.orchestra.run.smithi091.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 576 kB/s | 48 kB 00:00 2023-11-24T06:05:04.688 INFO:teuthology.orchestra.run.smithi091.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.5 MB/s | 87 kB 00:00 2023-11-24T06:05:04.713 INFO:teuthology.orchestra.run.smithi091.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 728 kB/s | 43 kB 00:00 2023-11-24T06:05:04.738 INFO:teuthology.orchestra.run.smithi091.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 2.1 MB/s | 109 kB 00:00 2023-11-24T06:05:04.763 INFO:teuthology.orchestra.run.smithi091.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.5 MB/s | 229 kB 00:00 2023-11-24T06:05:04.806 INFO:teuthology.orchestra.run.smithi091.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 8.2 MB/s | 560 kB 00:00 2023-11-24T06:05:04.831 INFO:teuthology.orchestra.run.smithi091.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 279 kB/s | 19 kB 00:00 2023-11-24T06:05:04.856 INFO:teuthology.orchestra.run.smithi091.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.0 MB/s | 155 kB 00:00 2023-11-24T06:05:04.890 INFO:teuthology.orchestra.run.smithi091.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 9.3 MB/s | 559 kB 00:00 2023-11-24T06:05:04.907 INFO:teuthology.orchestra.run.smithi091.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 356 kB/s | 18 kB 00:00 2023-11-24T06:05:04.949 INFO:teuthology.orchestra.run.smithi091.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 10 MB/s | 605 kB 00:00 2023-11-24T06:05:04.974 INFO:teuthology.orchestra.run.smithi091.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.5 MB/s | 238 kB 00:00 2023-11-24T06:05:05.016 INFO:teuthology.orchestra.run.smithi091.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.7 MB/s | 72 kB 00:00 2023-11-24T06:05:05.066 INFO:teuthology.orchestra.run.smithi091.stdout:(88/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 20 MB/s | 2.3 MB 00:00 2023-11-24T06:05:05.091 INFO:teuthology.orchestra.run.smithi091.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2023-11-24T06:05:05.142 INFO:teuthology.orchestra.run.smithi091.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 792 kB/s | 60 kB 00:00 2023-11-24T06:05:05.189 INFO:teuthology.orchestra.run.smithi091.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.0 MB/s | 98 kB 00:00 2023-11-24T06:05:05.218 INFO:teuthology.orchestra.run.smithi091.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 768 kB/s | 58 kB 00:00 2023-11-24T06:05:05.243 INFO:teuthology.orchestra.run.smithi091.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 726 kB/s | 39 kB 00:00 2023-11-24T06:05:05.285 INFO:teuthology.orchestra.run.smithi091.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 9.4 MB/s | 644 kB 00:00 2023-11-24T06:05:05.310 INFO:teuthology.orchestra.run.smithi091.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.5 MB/s | 171 kB 00:00 2023-11-24T06:05:05.335 INFO:teuthology.orchestra.run.smithi091.stdout:(96/131): NetworkManager-initscripts-updown-1.3 2.7 MB/s | 138 kB 00:00 2023-11-24T06:05:05.386 INFO:teuthology.orchestra.run.smithi091.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.2 MB/s | 110 kB 00:00 2023-11-24T06:05:05.436 INFO:teuthology.orchestra.run.smithi091.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2023-11-24T06:05:05.486 INFO:teuthology.orchestra.run.smithi091.stdout:(99/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2023-11-24T06:05:05.537 INFO:teuthology.orchestra.run.smithi091.stdout:(100/131): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-11-24T06:05:05.578 INFO:teuthology.orchestra.run.smithi091.stdout:(101/131): python3-markupsafe-0.23-19.el8.x86_6 934 kB/s | 39 kB 00:00 2023-11-24T06:05:05.759 INFO:teuthology.orchestra.run.smithi091.stdout:(102/131): python3-scipy-1.0.0-21.module+el8.5. 31 MB/s | 14 MB 00:00 2023-11-24T06:05:05.796 INFO:teuthology.orchestra.run.smithi091.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 3.1 MB/s | 688 kB 00:00 2023-11-24T06:05:05.829 INFO:teuthology.orchestra.run.smithi091.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 7.5 MB/s | 538 kB 00:00 2023-11-24T06:05:05.855 INFO:teuthology.orchestra.run.smithi091.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-11-24T06:05:05.880 INFO:teuthology.orchestra.run.smithi091.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-11-24T06:05:05.914 INFO:teuthology.orchestra.run.smithi091.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-11-24T06:05:05.964 INFO:teuthology.orchestra.run.smithi091.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 530 kB/s | 27 kB 00:00 2023-11-24T06:05:06.048 INFO:teuthology.orchestra.run.smithi091.stdout:(109/131): python3-babel-2.5.1-7.el8.noarch.rpm 29 MB/s | 4.8 MB 00:00 2023-11-24T06:05:06.072 INFO:teuthology.orchestra.run.smithi091.stdout:(110/131): flex-2.6.1-9.el8.x86_64.rpm 2.9 MB/s | 320 kB 00:00 2023-11-24T06:05:06.098 INFO:teuthology.orchestra.run.smithi091.stdout:(111/131): python3-prettytable-0.7.2-14.el8.noa 878 kB/s | 44 kB 00:00 2023-11-24T06:05:06.148 INFO:teuthology.orchestra.run.smithi091.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 9.0 MB/s | 457 kB 00:00 2023-11-24T06:05:06.215 INFO:teuthology.orchestra.run.smithi091.stdout:(113/131): python3-numpy-1.14.3-10.el8.x86_64.r 26 MB/s | 3.7 MB 00:00 2023-11-24T06:05:06.240 INFO:teuthology.orchestra.run.smithi091.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 2.4 MB/s | 225 kB 00:00 2023-11-24T06:05:06.265 INFO:teuthology.orchestra.run.smithi091.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2023-11-24T06:05:06.299 INFO:teuthology.orchestra.run.smithi091.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.1 MB/s | 126 kB 00:00 2023-11-24T06:05:06.349 INFO:teuthology.orchestra.run.smithi091.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 239 kB/s | 20 kB 00:00 2023-11-24T06:05:06.375 INFO:teuthology.orchestra.run.smithi091.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 1.3 MB/s | 103 kB 00:00 2023-11-24T06:05:06.417 INFO:teuthology.orchestra.run.smithi091.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2023-11-24T06:05:06.467 INFO:teuthology.orchestra.run.smithi091.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 3.9 MB/s | 198 kB 00:00 2023-11-24T06:05:06.551 INFO:teuthology.orchestra.run.smithi091.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 28 MB/s | 4.9 MB 00:00 2023-11-24T06:05:06.593 INFO:teuthology.orchestra.run.smithi091.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.7 MB/s | 202 kB 00:00 2023-11-24T06:05:06.668 INFO:teuthology.orchestra.run.smithi091.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2023-11-24T06:05:06.693 INFO:teuthology.orchestra.run.smithi091.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.4 MB/s | 354 kB 00:00 2023-11-24T06:05:06.761 INFO:teuthology.orchestra.run.smithi091.stdout:(125/131): postgresql-contrib-10.21-2.module+el 12 MB/s | 808 kB 00:00 2023-11-24T06:05:06.844 INFO:teuthology.orchestra.run.smithi091.stdout:(126/131): postgresql-server-10.21-2.module+el8 29 MB/s | 5.1 MB 00:00 2023-11-24T06:05:06.906 INFO:teuthology.orchestra.run.smithi091.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 10 MB/s | 1.5 MB 00:00 2023-11-24T06:05:07.778 INFO:teuthology.orchestra.run.smithi091.stdout:(128/131): librbd1-18.0.0-7484.ga13ea5ff.el8.x8 4.3 MB/s | 3.7 MB 00:00 2023-11-24T06:05:08.262 INFO:teuthology.orchestra.run.smithi091.stdout:(129/131): ceph-test-18.0.0-7484.ga13ea5ff.el8. 5.9 MB/s | 49 MB 00:08 2023-11-24T06:05:08.295 INFO:teuthology.orchestra.run.smithi091.stdout:(130/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 1.5 MB/s | 798 kB 00:00 2023-11-24T06:05:08.545 INFO:teuthology.orchestra.run.smithi091.stdout:(131/131): librados2-18.0.0-7484.ga13ea5ff.el8. 2.2 MB/s | 3.8 MB 00:01 2023-11-24T06:05:08.549 INFO:teuthology.orchestra.run.smithi091.stdout:-------------------------------------------------------------------------------- 2023-11-24T06:05:08.550 INFO:teuthology.orchestra.run.smithi091.stdout:Total 18 MB/s | 209 MB 00:11 2023-11-24T06:05:09.397 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction check 2023-11-24T06:05:09.517 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction check succeeded. 2023-11-24T06:05:09.517 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction test 2023-11-24T06:05:10.642 INFO:teuthology.orchestra.run.smithi091.stdout:Transaction test succeeded. 2023-11-24T06:05:10.646 INFO:teuthology.orchestra.run.smithi091.stdout:Running transaction 2023-11-24T06:05:12.536 INFO:teuthology.orchestra.run.smithi094.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-24T06:05:12.537 INFO:teuthology.orchestra.run.smithi094.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-11-24T06:05:12.551 INFO:teuthology.orchestra.run.smithi094.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-11-24T06:05:12.552 INFO:teuthology.orchestra.run.smithi094.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-11-24T06:05:12.553 INFO:teuthology.orchestra.run.smithi094.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-11-24T06:05:12.554 INFO:teuthology.orchestra.run.smithi094.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-11-24T06:05:12.556 INFO:teuthology.orchestra.run.smithi094.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-11-24T06:05:12.559 INFO:teuthology.orchestra.run.smithi094.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-11-24T06:05:12.560 INFO:teuthology.orchestra.run.smithi094.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-11-24T06:05:12.695 INFO:teuthology.orchestra.run.smithi094.stdout:Dependencies resolved. 2023-11-24T06:05:12.707 INFO:teuthology.orchestra.run.smithi094.stdout:=========================================================================================================================== 2023-11-24T06:05:12.707 INFO:teuthology.orchestra.run.smithi094.stdout: Package Arch Version Repository Size 2023-11-24T06:05:12.707 INFO:teuthology.orchestra.run.smithi094.stdout:=========================================================================================================================== 2023-11-24T06:05:12.707 INFO:teuthology.orchestra.run.smithi094.stdout:Installing: 2023-11-24T06:05:12.707 INFO:teuthology.orchestra.run.smithi094.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-11-24T06:05:12.707 INFO:teuthology.orchestra.run.smithi094.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-11-24T06:05:12.708 INFO:teuthology.orchestra.run.smithi094.stdout: ceph x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 6.4 k 2023-11-24T06:05:12.708 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 5.4 M 2023-11-24T06:05:12.708 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 920 k 2023-11-24T06:05:12.708 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 171 k 2023-11-24T06:05:12.708 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 1.6 M 2023-11-24T06:05:12.708 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 133 k 2023-11-24T06:05:12.708 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 1.8 M 2023-11-24T06:05:12.708 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 7.4 M 2023-11-24T06:05:12.708 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 50 k 2023-11-24T06:05:12.708 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 14 M 2023-11-24T06:05:12.708 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 49 M 2023-11-24T06:05:12.708 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 262 k 2023-11-24T06:05:12.709 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 407 k 2023-11-24T06:05:12.709 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-mirror x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 262 k 2023-11-24T06:05:12.709 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-top noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 18 k 2023-11-24T06:05:12.709 INFO:teuthology.orchestra.run.smithi094.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-11-24T06:05:12.709 INFO:teuthology.orchestra.run.smithi094.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-11-24T06:05:12.709 INFO:teuthology.orchestra.run.smithi094.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-11-24T06:05:12.709 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 31 k 2023-11-24T06:05:12.709 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 839 k 2023-11-24T06:05:12.709 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 139 k 2023-11-24T06:05:12.709 INFO:teuthology.orchestra.run.smithi094.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-11-24T06:05:12.709 INFO:teuthology.orchestra.run.smithi094.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-11-24T06:05:12.709 INFO:teuthology.orchestra.run.smithi094.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-11-24T06:05:12.709 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-11-24T06:05:12.710 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-11-24T06:05:12.710 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-11-24T06:05:12.710 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 197 k 2023-11-24T06:05:12.710 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 371 k 2023-11-24T06:05:12.710 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 364 k 2023-11-24T06:05:12.710 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 107 k 2023-11-24T06:05:12.710 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 94 k 2023-11-24T06:05:12.710 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 3.6 M 2023-11-24T06:05:12.710 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 181 k 2023-11-24T06:05:12.710 INFO:teuthology.orchestra.run.smithi094.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-11-24T06:05:12.710 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrading: 2023-11-24T06:05:12.710 INFO:teuthology.orchestra.run.smithi094.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-11-24T06:05:12.711 INFO:teuthology.orchestra.run.smithi094.stdout: librados2 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 3.8 M 2023-11-24T06:05:12.711 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 3.7 M 2023-11-24T06:05:12.711 INFO:teuthology.orchestra.run.smithi094.stdout:Installing dependencies: 2023-11-24T06:05:12.711 INFO:teuthology.orchestra.run.smithi094.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-11-24T06:05:12.711 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 23 M 2023-11-24T06:05:12.711 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 28 k 2023-11-24T06:05:12.711 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 2.3 M 2023-11-24T06:05:12.711 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 243 k 2023-11-24T06:05:12.711 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 4.6 M 2023-11-24T06:05:12.711 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 18 M 2023-11-24T06:05:12.711 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7484.ga13ea5ff.el8 ceph-noarch 15 k 2023-11-24T06:05:12.711 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 24 k 2023-11-24T06:05:12.711 INFO:teuthology.orchestra.run.smithi094.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-24T06:05:12.712 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-24T06:05:12.712 INFO:teuthology.orchestra.run.smithi094.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-24T06:05:12.712 INFO:teuthology.orchestra.run.smithi094.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-11-24T06:05:12.712 INFO:teuthology.orchestra.run.smithi094.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-11-24T06:05:12.712 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 183 k 2023-11-24T06:05:12.712 INFO:teuthology.orchestra.run.smithi094.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-11-24T06:05:12.712 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-24T06:05:12.712 INFO:teuthology.orchestra.run.smithi094.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-24T06:05:12.712 INFO:teuthology.orchestra.run.smithi094.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-11-24T06:05:12.712 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-24T06:05:12.712 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-24T06:05:12.712 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 517 k 2023-11-24T06:05:12.712 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-24T06:05:12.713 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2 x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 6.9 M 2023-11-24T06:05:12.713 INFO:teuthology.orchestra.run.smithi094.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-11-24T06:05:12.713 INFO:teuthology.orchestra.run.smithi094.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-11-24T06:05:12.713 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-24T06:05:12.713 INFO:teuthology.orchestra.run.smithi094.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-11-24T06:05:12.713 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-24T06:05:12.713 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-24T06:05:12.713 INFO:teuthology.orchestra.run.smithi094.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-24T06:05:12.713 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-24T06:05:12.713 INFO:teuthology.orchestra.run.smithi094.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-11-24T06:05:12.713 INFO:teuthology.orchestra.run.smithi094.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-11-24T06:05:12.713 INFO:teuthology.orchestra.run.smithi094.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-11-24T06:05:12.713 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-24T06:05:12.714 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-24T06:05:12.714 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-24T06:05:12.714 INFO:teuthology.orchestra.run.smithi094.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-24T06:05:12.714 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-24T06:05:12.714 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 45 k 2023-11-24T06:05:12.714 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common x86_64 2:18.0.0-7484.ga13ea5ff.el8 ceph 128 k 2023-11-24T06:05:12.714 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-24T06:05:12.714 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-24T06:05:12.714 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-24T06:05:12.714 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-24T06:05:12.714 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-24T06:05:12.715 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-24T06:05:12.715 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-24T06:05:12.715 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-24T06:05:12.715 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-24T06:05:12.715 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-24T06:05:12.715 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-24T06:05:12.715 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-24T06:05:12.715 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-24T06:05:12.715 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-24T06:05:12.715 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-24T06:05:12.715 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-24T06:05:12.715 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-24T06:05:12.716 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-24T06:05:12.716 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-24T06:05:12.716 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-24T06:05:12.716 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-24T06:05:12.716 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-24T06:05:12.716 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-24T06:05:12.716 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-24T06:05:12.716 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-24T06:05:12.716 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-24T06:05:12.716 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-24T06:05:12.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-24T06:05:12.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-24T06:05:12.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-24T06:05:12.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-24T06:05:12.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-24T06:05:12.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-24T06:05:12.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-24T06:05:12.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-24T06:05:12.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-24T06:05:12.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-24T06:05:12.717 INFO:teuthology.orchestra.run.smithi094.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-24T06:05:12.718 INFO:teuthology.orchestra.run.smithi094.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-24T06:05:12.718 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-24T06:05:12.718 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-24T06:05:12.718 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-24T06:05:12.718 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-24T06:05:12.718 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-24T06:05:12.718 INFO:teuthology.orchestra.run.smithi094.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-24T06:05:12.718 INFO:teuthology.orchestra.run.smithi094.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-24T06:05:12.718 INFO:teuthology.orchestra.run.smithi094.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-11-24T06:05:12.718 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-24T06:05:12.718 INFO:teuthology.orchestra.run.smithi094.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-11-24T06:05:12.719 INFO:teuthology.orchestra.run.smithi094.stdout:Enabling module streams: 2023-11-24T06:05:12.719 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql 10 2023-11-24T06:05:12.719 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:05:12.719 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction Summary 2023-11-24T06:05:12.719 INFO:teuthology.orchestra.run.smithi094.stdout:=========================================================================================================================== 2023-11-24T06:05:12.719 INFO:teuthology.orchestra.run.smithi094.stdout:Install 128 Packages 2023-11-24T06:05:12.719 INFO:teuthology.orchestra.run.smithi094.stdout:Upgrade 3 Packages 2023-11-24T06:05:12.719 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:05:12.720 INFO:teuthology.orchestra.run.smithi094.stdout:Total download size: 209 M 2023-11-24T06:05:12.720 INFO:teuthology.orchestra.run.smithi094.stdout:Downloading Packages: 2023-11-24T06:05:12.889 INFO:teuthology.orchestra.run.smithi094.stdout:(1/131): ceph-18.0.0-7484.ga13ea5ff.el8.x86_64. 47 kB/s | 6.4 kB 00:00 2023-11-24T06:05:13.139 INFO:teuthology.orchestra.run.smithi094.stdout:(2/131): ceph-fuse-18.0.0-7484.ga13ea5ff.el8.x8 3.6 MB/s | 920 kB 00:00 2023-11-24T06:05:13.206 INFO:teuthology.orchestra.run.smithi094.stdout:(3/131): ceph-immutable-object-cache-18.0.0-748 2.5 MB/s | 171 kB 00:00 2023-11-24T06:05:13.507 INFO:teuthology.orchestra.run.smithi094.stdout:(4/131): ceph-mds-18.0.0-7484.ga13ea5ff.el8.x86 7.7 MB/s | 2.3 MB 00:00 2023-11-24T06:05:13.707 INFO:teuthology.orchestra.run.smithi094.stdout:(5/131): ceph-mgr-18.0.0-7484.ga13ea5ff.el8.x86 8.1 MB/s | 1.6 MB 00:00 2023-11-24T06:05:14.199 INFO:teuthology.orchestra.run.smithi094.stdout:(6/131): ceph-mon-18.0.0-7484.ga13ea5ff.el8.x86 9.4 MB/s | 4.6 MB 00:00 2023-11-24T06:05:14.685 INFO:teuthology.orchestra.run.smithi094.stdout:(7/131): ceph-common-18.0.0-7484.ga13ea5ff.el8. 12 MB/s | 23 MB 00:01 2023-11-24T06:05:15.584 INFO:teuthology.orchestra.run.smithi091.stdout: Preparing : 1/1 2023-11-24T06:05:15.796 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-24T06:05:15.808 INFO:teuthology.orchestra.run.smithi094.stdout:(8/131): ceph-radosgw-18.0.0-7484.ga13ea5ff.el8 12 MB/s | 14 MB 00:01 2023-11-24T06:05:15.962 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-24T06:05:15.983 INFO:teuthology.orchestra.run.smithi094.stdout:(9/131): ceph-osd-18.0.0-7484.ga13ea5ff.el8.x86 10 MB/s | 18 MB 00:01 2023-11-24T06:05:16.000 INFO:teuthology.orchestra.run.smithi094.stdout:(10/131): ceph-selinux-18.0.0-7484.ga13ea5ff.el 127 kB/s | 24 kB 00:00 2023-11-24T06:05:16.050 INFO:teuthology.orchestra.run.smithi094.stdout:(11/131): cephfs-mirror-18.0.0-7484.ga13ea5ff.e 5.1 MB/s | 262 kB 00:00 2023-11-24T06:05:16.114 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-11-24T06:05:16.117 INFO:teuthology.orchestra.run.smithi094.stdout:(12/131): libcephfs-devel-18.0.0-7484.ga13ea5ff 465 kB/s | 31 kB 00:00 2023-11-24T06:05:16.244 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-11-24T06:05:16.259 INFO:teuthology.orchestra.run.smithi094.stdout:(13/131): libcephfs2-18.0.0-7484.ga13ea5ff.el8. 5.8 MB/s | 839 kB 00:00 2023-11-24T06:05:16.318 INFO:teuthology.orchestra.run.smithi094.stdout:(14/131): libcephsqlite-18.0.0-7484.ga13ea5ff.e 3.1 MB/s | 183 kB 00:00 2023-11-24T06:05:16.377 INFO:teuthology.orchestra.run.smithi094.stdout:(15/131): librados-devel-18.0.0-7484.ga13ea5ff. 2.3 MB/s | 139 kB 00:00 2023-11-24T06:05:16.448 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-11-24T06:05:16.452 INFO:teuthology.orchestra.run.smithi094.stdout:(16/131): libradosstriper1-18.0.0-7484.ga13ea5f 6.7 MB/s | 517 kB 00:00 2023-11-24T06:05:16.855 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-11-24T06:05:17.069 INFO:teuthology.orchestra.run.smithi094.stdout:(17/131): librgw2-18.0.0-7484.ga13ea5ff.el8.x86 11 MB/s | 6.9 MB 00:00 2023-11-24T06:05:17.097 INFO:teuthology.orchestra.run.smithi091.stdout: Upgrading : librados2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 5/134 2023-11-24T06:05:17.144 INFO:teuthology.orchestra.run.smithi094.stdout:(18/131): python3-ceph-argparse-18.0.0-7484.ga1 599 kB/s | 45 kB 00:00 2023-11-24T06:05:17.211 INFO:teuthology.orchestra.run.smithi094.stdout:(19/131): python3-ceph-common-18.0.0-7484.ga13e 1.9 MB/s | 128 kB 00:00 2023-11-24T06:05:17.278 INFO:teuthology.orchestra.run.smithi094.stdout:(20/131): python3-cephfs-18.0.0-7484.ga13ea5ff. 2.9 MB/s | 197 kB 00:00 2023-11-24T06:05:17.354 INFO:teuthology.orchestra.run.smithi094.stdout:(21/131): python3-rados-18.0.0-7484.ga13ea5ff.e 4.8 MB/s | 371 kB 00:00 2023-11-24T06:05:17.438 INFO:teuthology.orchestra.run.smithi094.stdout:(22/131): python3-rbd-18.0.0-7484.ga13ea5ff.el8 4.3 MB/s | 364 kB 00:00 2023-11-24T06:05:17.505 INFO:teuthology.orchestra.run.smithi094.stdout:(23/131): python3-rgw-18.0.0-7484.ga13ea5ff.el8 1.6 MB/s | 107 kB 00:00 2023-11-24T06:05:17.507 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: librados2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 5/134 2023-11-24T06:05:17.572 INFO:teuthology.orchestra.run.smithi094.stdout:(24/131): rbd-fuse-18.0.0-7484.ga13ea5ff.el8.x8 1.4 MB/s | 94 kB 00:00 2023-11-24T06:05:17.714 INFO:teuthology.orchestra.run.smithi091.stdout: Upgrading : librbd1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 6/134 2023-11-24T06:05:17.849 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: librbd1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 6/134 2023-11-24T06:05:17.913 INFO:teuthology.orchestra.run.smithi094.stdout:(25/131): rbd-mirror-18.0.0-7484.ga13ea5ff.el8. 10 MB/s | 3.6 MB 00:00 2023-11-24T06:05:17.981 INFO:teuthology.orchestra.run.smithi094.stdout:(26/131): rbd-nbd-18.0.0-7484.ga13ea5ff.el8.x86 2.7 MB/s | 181 kB 00:00 2023-11-24T06:05:18.047 INFO:teuthology.orchestra.run.smithi094.stdout:(27/131): ceph-grafana-dashboards-18.0.0-7484.g 420 kB/s | 28 kB 00:00 2023-11-24T06:05:18.056 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libcephfs2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 7/134 2023-11-24T06:05:18.114 INFO:teuthology.orchestra.run.smithi094.stdout:(28/131): ceph-mgr-cephadm-18.0.0-7484.ga13ea5f 1.9 MB/s | 133 kB 00:00 2023-11-24T06:05:18.133 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: libcephfs2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 7/134 2023-11-24T06:05:18.269 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-rados-2:18.0.0-7484.ga13ea5ff.el8.x86_64 8/134 2023-11-24T06:05:18.298 INFO:teuthology.orchestra.run.smithi094.stdout:(29/131): ceph-mgr-dashboard-18.0.0-7484.ga13ea 9.6 MB/s | 1.8 MB 00:00 2023-11-24T06:05:18.432 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-11-24T06:05:18.684 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-24T06:05:18.875 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-24T06:05:19.015 INFO:teuthology.orchestra.run.smithi094.stdout:(30/131): ceph-test-18.0.0-7484.ga13ea5ff.el8.x 16 MB/s | 49 MB 00:03 2023-11-24T06:05:19.066 INFO:teuthology.orchestra.run.smithi094.stdout:(31/131): ceph-mgr-modules-core-18.0.0-7484.ga1 4.8 MB/s | 243 kB 00:00 2023-11-24T06:05:19.174 INFO:teuthology.orchestra.run.smithi094.stdout:(32/131): ceph-mgr-diskprediction-local-18.0.0- 8.5 MB/s | 7.4 MB 00:00 2023-11-24T06:05:19.191 INFO:teuthology.orchestra.run.smithi094.stdout:(33/131): ceph-mgr-rook-18.0.0-7484.ga13ea5ff.e 400 kB/s | 50 kB 00:00 2023-11-24T06:05:19.216 INFO:teuthology.orchestra.run.smithi094.stdout:(34/131): ceph-prometheus-alerts-18.0.0-7484.ga 358 kB/s | 15 kB 00:00 2023-11-24T06:05:19.242 INFO:teuthology.orchestra.run.smithi094.stdout:(35/131): ceph-volume-18.0.0-7484.ga13ea5ff.el8 5.1 MB/s | 262 kB 00:00 2023-11-24T06:05:19.267 INFO:teuthology.orchestra.run.smithi094.stdout:(36/131): cephadm-18.0.0-7484.ga13ea5ff.el8.noa 7.9 MB/s | 407 kB 00:00 2023-11-24T06:05:19.268 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-24T06:05:19.284 INFO:teuthology.orchestra.run.smithi094.stdout:(37/131): cephfs-top-18.0.0-7484.ga13ea5ff.el8. 431 kB/s | 18 kB 00:00 2023-11-24T06:05:19.496 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-24T06:05:19.655 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libcephsqlite-2:18.0.0-7484.ga13ea5ff.el8.x86_64 14/134 2023-11-24T06:05:19.743 INFO:teuthology.orchestra.run.smithi094.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 128 kB/s | 59 kB 00:00 2023-11-24T06:05:19.784 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7484.ga13ea5ff.el8.x86_64 14/134 2023-11-24T06:05:19.939 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libradosstriper1-2:18.0.0-7484.ga13ea5ff.el8.x86 15/134 2023-11-24T06:05:20.002 INFO:teuthology.orchestra.run.smithi094.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 290 kB/s | 75 kB 00:00 2023-11-24T06:05:20.066 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7484.ga13ea5ff.el8.x86 15/134 2023-11-24T06:05:20.242 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-11-24T06:05:20.436 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-11-24T06:05:20.493 INFO:teuthology.orchestra.run.smithi094.stdout:(40/131): gperftools-libs-2.7-9.el8.x86_64.rpm 250 kB/s | 306 kB 00:01 2023-11-24T06:05:20.586 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-11-24T06:05:20.644 INFO:teuthology.orchestra.run.smithi094.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 290 kB/s | 44 kB 00:00 2023-11-24T06:05:20.713 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-11-24T06:05:20.833 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-11-24T06:05:21.047 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-11-24T06:05:21.136 INFO:teuthology.orchestra.run.smithi094.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 377 kB/s | 185 kB 00:00 2023-11-24T06:05:21.403 INFO:teuthology.orchestra.run.smithi094.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 124 kB/s | 33 kB 00:00 2023-11-24T06:05:21.545 INFO:teuthology.orchestra.run.smithi094.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 115 kB/s | 16 kB 00:00 2023-11-24T06:05:21.671 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-11-24T06:05:21.787 INFO:teuthology.orchestra.run.smithi094.stdout:(45/131): python3-asyncssh-2.7.0-2.el8.noarch.r 266 kB/s | 476 kB 00:01 2023-11-24T06:05:21.830 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-11-24T06:05:21.908 INFO:teuthology.orchestra.run.smithi094.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 477 kB/s | 173 kB 00:00 2023-11-24T06:05:22.238 INFO:teuthology.orchestra.run.smithi094.stdout:(47/131): ceph-base-18.0.0-7484.ga13ea5ff.el8.x 580 kB/s | 5.4 MB 00:09 2023-11-24T06:05:22.263 INFO:teuthology.orchestra.run.smithi094.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 229 kB/s | 81 kB 00:00 2023-11-24T06:05:22.405 INFO:teuthology.orchestra.run.smithi094.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 124 kB/s | 18 kB 00:00 2023-11-24T06:05:22.428 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-11-24T06:05:22.489 INFO:teuthology.orchestra.run.smithi094.stdout:(50/131): python3-jaraco-6.2-6.el8.noarch.rpm 44 kB/s | 11 kB 00:00 2023-11-24T06:05:22.655 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-11-24T06:05:22.747 INFO:teuthology.orchestra.run.smithi094.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 188 kB/s | 49 kB 00:00 2023-11-24T06:05:22.998 INFO:teuthology.orchestra.run.smithi094.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 235 kB/s | 59 kB 00:00 2023-11-24T06:05:23.156 INFO:teuthology.orchestra.run.smithi094.stdout:(53/131): python3-cherrypy-18.4.0-1.el8.noarch. 281 kB/s | 384 kB 00:01 2023-11-24T06:05:23.257 INFO:teuthology.orchestra.run.smithi094.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 233 kB/s | 60 kB 00:00 2023-11-24T06:05:23.351 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-11-24T06:05:23.399 INFO:teuthology.orchestra.run.smithi094.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 110 kB/s | 16 kB 00:00 2023-11-24T06:05:23.549 INFO:teuthology.orchestra.run.smithi094.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 224 kB/s | 34 kB 00:00 2023-11-24T06:05:23.603 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-11-24T06:05:23.835 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-11-24T06:05:24.049 INFO:teuthology.orchestra.run.smithi094.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 393 kB/s | 196 kB 00:00 2023-11-24T06:05:24.056 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-24T06:05:24.175 INFO:teuthology.orchestra.run.smithi094.stdout:(58/131): python3-pecan-1.3.2-9.el8.noarch.rpm 278 kB/s | 283 kB 00:01 2023-11-24T06:05:24.227 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el8. 26/134 2023-11-24T06:05:24.325 INFO:teuthology.orchestra.run.smithi094.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 125 kB/s | 19 kB 00:00 2023-11-24T06:05:24.390 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el 27/134 2023-11-24T06:05:24.392 INFO:teuthology.orchestra.run.smithi094.stdout:(60/131): python3-rsa-4.9-2.el8.noarch.rpm 187 kB/s | 64 kB 00:00 2023-11-24T06:05:24.459 INFO:teuthology.orchestra.run.smithi094.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 178 kB/s | 24 kB 00:00 2023-11-24T06:05:24.526 INFO:teuthology.orchestra.run.smithi094.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 219 kB/s | 29 kB 00:00 2023-11-24T06:05:24.594 INFO:teuthology.orchestra.run.smithi094.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 205 kB/s | 27 kB 00:00 2023-11-24T06:05:24.602 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-cephfs-2:18.0.0-7484.ga13ea5ff.el8.x86_6 28/134 2023-11-24T06:05:24.644 INFO:teuthology.orchestra.run.smithi094.stdout:(64/131): python3-kubernetes-11.0.0-6.el8.noarc 533 kB/s | 1.2 MB 00:02 2023-11-24T06:05:24.819 INFO:teuthology.orchestra.run.smithi094.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 350 kB/s | 61 kB 00:00 2023-11-24T06:05:24.821 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-11-24T06:05:25.078 INFO:teuthology.orchestra.run.smithi094.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 330 kB/s | 85 kB 00:00 2023-11-24T06:05:25.145 INFO:teuthology.orchestra.run.smithi094.stdout:(67/131): python3-waitress-1.4.3-1.el8.noarch.r 413 kB/s | 255 kB 00:00 2023-11-24T06:05:25.220 INFO:teuthology.orchestra.run.smithi094.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 161 kB/s | 23 kB 00:00 2023-11-24T06:05:25.245 INFO:teuthology.orchestra.run.smithi094.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-11-24T06:05:25.546 INFO:teuthology.orchestra.run.smithi094.stdout:(70/131): python3-scipy-1.0.0-21.module+el8.5.0 46 MB/s | 14 MB 00:00 2023-11-24T06:05:25.571 INFO:teuthology.orchestra.run.smithi094.stdout:(71/131): python3-webob-1.8.5-1.el8.1.noarch.rp 257 kB/s | 251 kB 00:00 2023-11-24T06:05:25.597 INFO:teuthology.orchestra.run.smithi094.stdout:(72/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.1 MB/s | 110 kB 00:00 2023-11-24T06:05:25.638 INFO:teuthology.orchestra.run.smithi094.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 810 kB/s | 54 kB 00:00 2023-11-24T06:05:25.664 INFO:teuthology.orchestra.run.smithi094.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.1 MB/s | 259 kB 00:00 2023-11-24T06:05:25.689 INFO:teuthology.orchestra.run.smithi094.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-11-24T06:05:25.714 INFO:teuthology.orchestra.run.smithi094.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 769 kB/s | 39 kB 00:00 2023-11-24T06:05:25.764 INFO:teuthology.orchestra.run.smithi094.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 8.9 MB/s | 688 kB 00:00 2023-11-24T06:05:25.807 INFO:teuthology.orchestra.run.smithi094.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 5.7 MB/s | 538 kB 00:00 2023-11-24T06:05:25.832 INFO:teuthology.orchestra.run.smithi094.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.3 MB/s | 157 kB 00:00 2023-11-24T06:05:25.857 INFO:teuthology.orchestra.run.smithi094.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-11-24T06:05:25.899 INFO:teuthology.orchestra.run.smithi094.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 10 MB/s | 713 kB 00:00 2023-11-24T06:05:25.933 INFO:teuthology.orchestra.run.smithi094.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 802 kB/s | 27 kB 00:00 2023-11-24T06:05:25.975 INFO:teuthology.orchestra.run.smithi094.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-11-24T06:05:26.050 INFO:teuthology.orchestra.run.smithi094.stdout:(84/131): python3-babel-2.5.1-7.el8.noarch.rpm 25 MB/s | 4.8 MB 00:00 2023-11-24T06:05:26.067 INFO:teuthology.orchestra.run.smithi094.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 478 kB/s | 44 kB 00:00 2023-11-24T06:05:26.109 INFO:teuthology.orchestra.run.smithi094.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 11 MB/s | 457 kB 00:00 2023-11-24T06:05:26.193 INFO:teuthology.orchestra.run.smithi094.stdout:(87/131): python3-numpy-1.14.3-10.el8.x86_64.rp 26 MB/s | 3.7 MB 00:00 2023-11-24T06:05:26.218 INFO:teuthology.orchestra.run.smithi094.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 2.0 MB/s | 225 kB 00:00 2023-11-24T06:05:26.244 INFO:teuthology.orchestra.run.smithi094.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2023-11-24T06:05:26.269 INFO:teuthology.orchestra.run.smithi094.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-11-24T06:05:26.294 INFO:teuthology.orchestra.run.smithi094.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 399 kB/s | 20 kB 00:00 2023-11-24T06:05:26.320 INFO:teuthology.orchestra.run.smithi094.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2023-11-24T06:05:26.362 INFO:teuthology.orchestra.run.smithi094.stdout:(93/131): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2023-11-24T06:05:26.546 INFO:teuthology.orchestra.run.smithi094.stdout:(94/131): libpq-13.5-1.el8.x86_64.rpm 1.1 MB/s | 198 kB 00:00 2023-11-24T06:05:26.601 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-11-24T06:05:26.646 INFO:teuthology.orchestra.run.smithi094.stdout:(95/131): openblas-threads-0.3.15-3.el8.x86_64. 15 MB/s | 4.9 MB 00:00 2023-11-24T06:05:26.688 INFO:teuthology.orchestra.run.smithi094.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.7 MB/s | 202 kB 00:00 2023-11-24T06:05:26.763 INFO:teuthology.orchestra.run.smithi094.stdout:(97/131): openblas-0.3.15-3.el8.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2023-11-24T06:05:26.789 INFO:teuthology.orchestra.run.smithi094.stdout:(98/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.4 MB/s | 354 kB 00:00 2023-11-24T06:05:26.839 INFO:teuthology.orchestra.run.smithi094.stdout:(99/131): postgresql-contrib-10.21-2.module+el8 16 MB/s | 808 kB 00:00 2023-11-24T06:05:26.931 INFO:teuthology.orchestra.run.smithi094.stdout:(100/131): postgresql-server-10.21-2.module+el8 30 MB/s | 5.1 MB 00:00 2023-11-24T06:05:26.965 INFO:teuthology.orchestra.run.smithi094.stdout:(101/131): postgresql-10.21-2.module+el8.6.0+15 12 MB/s | 1.5 MB 00:00 2023-11-24T06:05:26.990 INFO:teuthology.orchestra.run.smithi094.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 798 kB/s | 47 kB 00:00 2023-11-24T06:05:27.041 INFO:teuthology.orchestra.run.smithi094.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 2.6 MB/s | 200 kB 00:00 2023-11-24T06:05:27.082 INFO:teuthology.orchestra.run.smithi094.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 462 kB/s | 43 kB 00:00 2023-11-24T06:05:27.108 INFO:teuthology.orchestra.run.smithi094.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2023-11-24T06:05:27.133 INFO:teuthology.orchestra.run.smithi094.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 952 kB/s | 48 kB 00:00 2023-11-24T06:05:27.158 INFO:teuthology.orchestra.run.smithi094.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-11-24T06:05:27.175 INFO:teuthology.orchestra.run.smithi094.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 1.0 MB/s | 43 kB 00:00 2023-11-24T06:05:27.201 INFO:teuthology.orchestra.run.smithi094.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.5 MB/s | 109 kB 00:00 2023-11-24T06:05:27.226 INFO:teuthology.orchestra.run.smithi094.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.4 MB/s | 229 kB 00:00 2023-11-24T06:05:27.260 INFO:teuthology.orchestra.run.smithi094.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.3 MB/s | 560 kB 00:00 2023-11-24T06:05:27.285 INFO:teuthology.orchestra.run.smithi094.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 318 kB/s | 19 kB 00:00 2023-11-24T06:05:27.311 INFO:teuthology.orchestra.run.smithi094.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.0 MB/s | 155 kB 00:00 2023-11-24T06:05:27.344 INFO:teuthology.orchestra.run.smithi094.stdout:(114/131): python3-cryptography-3.2.1-5.el8.x86 9.3 MB/s | 559 kB 00:00 2023-11-24T06:05:27.370 INFO:teuthology.orchestra.run.smithi094.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 305 kB/s | 18 kB 00:00 2023-11-24T06:05:27.412 INFO:teuthology.orchestra.run.smithi094.stdout:(116/131): pcre2-devel-10.32-2.el8.x86_64.rpm 8.8 MB/s | 605 kB 00:00 2023-11-24T06:05:27.437 INFO:teuthology.orchestra.run.smithi094.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.5 MB/s | 238 kB 00:00 2023-11-24T06:05:27.471 INFO:teuthology.orchestra.run.smithi094.stdout:(118/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 2.1 MB/s | 72 kB 00:00 2023-11-24T06:05:27.521 INFO:teuthology.orchestra.run.smithi094.stdout:(119/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 21 MB/s | 2.3 MB 00:00 2023-11-24T06:05:27.538 INFO:teuthology.orchestra.run.smithi094.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2023-11-24T06:05:27.563 INFO:teuthology.orchestra.run.smithi094.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.4 MB/s | 60 kB 00:00 2023-11-24T06:05:27.589 INFO:teuthology.orchestra.run.smithi094.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-11-24T06:05:27.614 INFO:teuthology.orchestra.run.smithi094.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-11-24T06:05:27.634 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-11-24T06:05:27.639 INFO:teuthology.orchestra.run.smithi094.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 764 kB/s | 39 kB 00:00 2023-11-24T06:05:27.682 INFO:teuthology.orchestra.run.smithi094.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 9.4 MB/s | 644 kB 00:00 2023-11-24T06:05:27.699 INFO:teuthology.orchestra.run.smithi094.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.8 MB/s | 171 kB 00:00 2023-11-24T06:05:27.724 INFO:teuthology.orchestra.run.smithi094.stdout:(127/131): NetworkManager-initscripts-updown-1. 3.2 MB/s | 138 kB 00:00 2023-11-24T06:05:27.970 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-11-24T06:05:27.999 INFO:teuthology.orchestra.run.smithi094.stdout:(128/131): librados2-18.0.0-7484.ga13ea5ff.el8. 13 MB/s | 3.8 MB 00:00 2023-11-24T06:05:28.075 INFO:teuthology.orchestra.run.smithi094.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 10 MB/s | 798 kB 00:00 2023-11-24T06:05:28.213 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-11-24T06:05:28.466 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-11-24T06:05:28.625 INFO:teuthology.orchestra.run.smithi094.stdout:(130/131): thrift-0.13.0-2.el8.x86_64.rpm 510 kB/s | 1.7 MB 00:03 2023-11-24T06:05:28.673 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-24T06:05:28.864 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-24T06:05:29.059 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-rbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 36/134 2023-11-24T06:05:29.193 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : librados-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_6 37/134 2023-11-24T06:05:29.319 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-11-24T06:05:29.446 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-11-24T06:05:29.628 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-11-24T06:05:29.691 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-11-24T06:05:29.904 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-11-24T06:05:30.156 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-11-24T06:05:30.303 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-11-24T06:05:30.386 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-11-24T06:05:30.634 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-11-24T06:05:30.841 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-11-24T06:05:30.986 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-11-24T06:05:31.155 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-11-24T06:05:31.356 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-11-24T06:05:31.498 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-11-24T06:05:31.566 INFO:teuthology.orchestra.run.smithi094.stdout:(131/131): librbd1-18.0.0-7484.ga13ea5ff.el8.x8 989 kB/s | 3.7 MB 00:03 2023-11-24T06:05:31.569 INFO:teuthology.orchestra.run.smithi094.stdout:-------------------------------------------------------------------------------- 2023-11-24T06:05:31.569 INFO:teuthology.orchestra.run.smithi094.stdout:Total 11 MB/s | 209 MB 00:18 2023-11-24T06:05:31.677 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-11-24T06:05:32.261 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-11-24T06:05:32.394 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction check 2023-11-24T06:05:32.514 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction check succeeded. 2023-11-24T06:05:32.515 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction test 2023-11-24T06:05:32.650 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-11-24T06:05:32.913 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-11-24T06:05:33.139 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-11-24T06:05:33.378 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-11-24T06:05:33.601 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-11-24T06:05:33.642 INFO:teuthology.orchestra.run.smithi094.stdout:Transaction test succeeded. 2023-11-24T06:05:33.646 INFO:teuthology.orchestra.run.smithi094.stdout:Running transaction 2023-11-24T06:05:33.900 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-11-24T06:05:34.143 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-11-24T06:05:34.329 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-11-24T06:05:34.533 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-11-24T06:05:34.733 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-11-24T06:05:34.898 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-11-24T06:05:35.188 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-11-24T06:05:35.466 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-11-24T06:05:35.690 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-11-24T06:05:35.961 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-11-24T06:05:36.118 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-24T06:05:36.296 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-24T06:05:36.585 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-24T06:05:36.792 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-24T06:05:36.939 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-24T06:05:37.128 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-24T06:05:37.308 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-24T06:05:37.495 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-24T06:05:37.702 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-24T06:05:37.846 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-24T06:05:37.982 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-24T06:05:38.234 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-24T06:05:38.429 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-24T06:05:38.486 INFO:teuthology.orchestra.run.smithi094.stdout: Preparing : 1/1 2023-11-24T06:05:38.606 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-24T06:05:38.671 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-11-24T06:05:38.838 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-11-24T06:05:38.867 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-24T06:05:39.046 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-11-24T06:05:39.091 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-24T06:05:39.206 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-24T06:05:39.248 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-24T06:05:39.336 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-11-24T06:05:39.472 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-24T06:05:39.565 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-24T06:05:39.973 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-11-24T06:05:40.206 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librados2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 6/134 2023-11-24T06:05:40.540 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-24T06:05:40.607 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 6/134 2023-11-24T06:05:40.779 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-24T06:05:40.814 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : librbd1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 7/134 2023-11-24T06:05:40.942 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 7/134 2023-11-24T06:05:41.017 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-24T06:05:41.123 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 8/134 2023-11-24T06:05:41.200 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephfs2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 8/134 2023-11-24T06:05:41.309 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-24T06:05:41.407 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rados-2:18.0.0-7484.ga13ea5ff.el8.x86_64 9/134 2023-11-24T06:05:41.583 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-24T06:05:41.692 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-11-24T06:05:41.820 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-24T06:05:41.892 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-11-24T06:05:42.300 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-11-24T06:05:42.571 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-11-24T06:05:42.631 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-24T06:05:42.825 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : librgw2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 92/134 2023-11-24T06:05:42.895 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: librgw2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 92/134 2023-11-24T06:05:42.919 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephsqlite-2:18.0.0-7484.ga13ea5ff.el8.x86_64 14/134 2023-11-24T06:05:43.054 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7484.ga13ea5ff.el8.x86_64 14/134 2023-11-24T06:05:43.068 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : python3-rgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 93/134 2023-11-24T06:05:43.231 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libradosstriper1-2:18.0.0-7484.ga13ea5ff.el8.x86 15/134 2023-11-24T06:05:43.311 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7484.ga13ea5ff.el8.x86 15/134 2023-11-24T06:05:43.439 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-24T06:05:43.575 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-11-24T06:05:43.722 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-24T06:05:44.343 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-11-24T06:05:44.497 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-24T06:05:45.109 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-11-24T06:05:45.306 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-11-24T06:05:46.009 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-11-24T06:05:46.271 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-11-24T06:05:46.497 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-11-24T06:05:46.734 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-11-24T06:05:46.748 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 94/134 2023-11-24T06:05:46.894 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-11-24T06:05:46.992 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 94/134 2023-11-24T06:05:47.102 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-11-24T06:05:47.357 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-11-24T06:05:47.545 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el8. 26/134 2023-11-24T06:05:47.592 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 94/134 2023-11-24T06:05:47.733 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el 27/134 2023-11-24T06:05:48.329 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-base-2:18.0.0-7484.ga13ea5ff.el8.x86_64 95/134 2023-11-24T06:05:48.414 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-base-2:18.0.0-7484.ga13ea5ff.el8.x86_64 95/134 2023-11-24T06:05:48.576 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-selinux-2:18.0.0-7484.ga13ea5ff.el8.x86_64 96/134 2023-11-24T06:05:48.641 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cephfs-2:18.0.0-7484.ga13ea5ff.el8.x86_6 28/134 2023-11-24T06:05:48.986 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-11-24T06:05:49.232 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-11-24T06:05:49.485 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-11-24T06:05:49.705 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-24T06:05:51.470 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-11-24T06:05:51.806 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-11-24T06:05:52.016 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-11-24T06:05:52.215 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-11-24T06:05:52.385 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 36/134 2023-11-24T06:05:52.550 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librados-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_6 37/134 2023-11-24T06:05:52.728 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-11-24T06:05:52.947 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-11-24T06:05:53.124 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-11-24T06:05:53.275 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-11-24T06:05:53.463 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-11-24T06:05:53.661 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-11-24T06:05:54.021 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-11-24T06:05:54.291 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-11-24T06:05:54.506 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-11-24T06:05:54.725 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-11-24T06:05:54.967 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-11-24T06:05:55.272 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-11-24T06:05:55.529 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-11-24T06:05:55.719 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-11-24T06:05:55.963 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-11-24T06:05:56.136 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-11-24T06:05:56.316 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-11-24T06:05:56.634 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-11-24T06:05:56.876 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-11-24T06:05:57.117 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-11-24T06:05:57.344 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-11-24T06:05:57.520 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-11-24T06:05:57.673 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-11-24T06:05:57.884 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-11-24T06:05:57.973 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-11-24T06:05:58.191 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-11-24T06:05:58.399 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-11-24T06:05:58.562 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-24T06:05:58.625 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-11-24T06:05:58.859 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-11-24T06:05:59.064 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-11-24T06:05:59.237 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-11-24T06:05:59.418 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-11-24T06:05:59.696 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-11-24T06:05:59.944 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-11-24T06:06:00.092 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-11-24T06:06:00.249 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-11-24T06:06:00.448 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-11-24T06:06:00.642 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-11-24T06:06:00.833 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-11-24T06:06:01.023 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-11-24T06:06:01.186 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-11-24T06:06:01.385 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-11-24T06:06:01.583 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-11-24T06:06:01.768 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-11-24T06:06:02.025 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-11-24T06:06:02.253 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-11-24T06:06:02.445 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-11-24T06:06:02.662 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-11-24T06:06:03.757 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-11-24T06:06:04.016 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-11-24T06:06:04.269 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-11-24T06:06:04.553 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-11-24T06:06:04.817 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-11-24T06:06:05.008 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-11-24T06:06:05.870 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-11-24T06:06:06.076 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : librgw2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 92/134 2023-11-24T06:06:06.141 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librgw2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 92/134 2023-11-24T06:06:06.321 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : python3-rgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 93/134 2023-11-24T06:06:08.305 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7484.ga13ea5ff.el8.x86_64 96/134 2023-11-24T06:06:08.306 INFO:teuthology.orchestra.run.smithi091.stdout:skipping the directory /sys 2023-11-24T06:06:08.306 INFO:teuthology.orchestra.run.smithi091.stdout:skipping the directory /proc 2023-11-24T06:06:08.306 INFO:teuthology.orchestra.run.smithi091.stdout:skipping the directory /dev 2023-11-24T06:06:08.306 INFO:teuthology.orchestra.run.smithi091.stdout:skipping the directory /run 2023-11-24T06:06:08.306 INFO:teuthology.orchestra.run.smithi091.stdout:skipping the directory /mnt 2023-11-24T06:06:08.306 INFO:teuthology.orchestra.run.smithi091.stdout:skipping the directory /var/tmp 2023-11-24T06:06:08.306 INFO:teuthology.orchestra.run.smithi091.stdout:skipping the directory /home 2023-11-24T06:06:08.306 INFO:teuthology.orchestra.run.smithi091.stdout:skipping the directory /tmp 2023-11-24T06:06:08.306 INFO:teuthology.orchestra.run.smithi091.stdout:skipping the directory /dev 2023-11-24T06:06:08.306 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:06:08.637 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-volume-2:18.0.0-7484.ga13ea5ff.el8.noarch 97/134 2023-11-24T06:06:09.979 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 94/134 2023-11-24T06:06:10.220 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 94/134 2023-11-24T06:06:10.232 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-volume-2:18.0.0-7484.ga13ea5ff.el8.noarch 97/134 2023-11-24T06:06:10.441 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-osd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 98/134 2023-11-24T06:06:10.667 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-osd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 98/134 2023-11-24T06:06:10.802 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 94/134 2023-11-24T06:06:10.837 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-mds-2:18.0.0-7484.ga13ea5ff.el8.x86_64 99/134 2023-11-24T06:06:11.080 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-base-2:18.0.0-7484.ga13ea5ff.el8.x86_64 95/134 2023-11-24T06:06:11.168 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-base-2:18.0.0-7484.ga13ea5ff.el8.x86_64 95/134 2023-11-24T06:06:11.293 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-mds-2:18.0.0-7484.ga13ea5ff.el8.x86_64 99/134 2023-11-24T06:06:11.336 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-selinux-2:18.0.0-7484.ga13ea5ff.el8.x86_64 96/134 2023-11-24T06:06:11.511 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-mon-2:18.0.0-7484.ga13ea5ff.el8.x86_64 100/134 2023-11-24T06:06:11.528 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-mon-2:18.0.0-7484.ga13ea5ff.el8.x86_64 100/134 2023-11-24T06:06:13.204 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 101/134 2023-11-24T06:06:13.418 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 101/134 2023-11-24T06:06:13.543 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.e 102/134 2023-11-24T06:06:13.726 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff. 103/134 2023-11-24T06:06:13.944 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el8.noa 104/134 2023-11-24T06:06:14.166 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el8.noa 104/134 2023-11-24T06:06:14.456 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el8.n 105/134 2023-11-24T06:06:15.413 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el8.n 105/134 2023-11-24T06:06:15.576 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 106/134 2023-11-24T06:06:15.668 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 106/134 2023-11-24T06:06:16.065 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el 107/134 2023-11-24T06:06:16.256 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-mgr-2:18.0.0-7484.ga13ea5ff.el8.x86_64 108/134 2023-11-24T06:06:16.293 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7484.ga13ea5ff.el8.x86_64 108/134 2023-11-24T06:06:16.485 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el8.noarch 109/134 2023-11-24T06:06:16.496 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el8.noarch 109/134 2023-11-24T06:06:17.811 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-2:18.0.0-7484.ga13ea5ff.el8.x86_64 110/134 2023-11-24T06:06:18.070 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 111/134 2023-11-24T06:06:18.104 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 111/134 2023-11-24T06:06:18.268 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 112/134 2023-11-24T06:06:18.310 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 112/134 2023-11-24T06:06:18.460 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 113/134 2023-11-24T06:06:18.818 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 113/134 2023-11-24T06:06:18.977 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : rbd-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 114/134 2023-11-24T06:06:23.642 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 114/134 2023-11-24T06:06:24.551 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-test-2:18.0.0-7484.ga13ea5ff.el8.x86_64 115/134 2023-11-24T06:06:24.871 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-11-24T06:06:25.074 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : ceph-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 117/134 2023-11-24T06:06:25.336 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-11-24T06:06:25.474 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-11-24T06:06:25.688 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-11-24T06:06:25.912 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-11-24T06:06:26.112 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_ 121/134 2023-11-24T06:06:26.242 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-24T06:06:28.395 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-24T06:06:28.641 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-24T06:06:28.661 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-24T06:06:28.799 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : cephfs-top-2:18.0.0-7484.ga13ea5ff.el8.noarch 124/134 2023-11-24T06:06:29.107 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : rbd-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 125/134 2023-11-24T06:06:29.404 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : rbd-nbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 126/134 2023-11-24T06:06:29.797 INFO:teuthology.orchestra.run.smithi091.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-24T06:06:29.828 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7484.ga13ea5ff.el8.x86_64 96/134 2023-11-24T06:06:29.829 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /sys 2023-11-24T06:06:29.829 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /proc 2023-11-24T06:06:29.829 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /dev 2023-11-24T06:06:29.829 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /run 2023-11-24T06:06:29.829 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /mnt 2023-11-24T06:06:29.829 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /var/tmp 2023-11-24T06:06:29.829 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /home 2023-11-24T06:06:29.829 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /tmp 2023-11-24T06:06:29.829 INFO:teuthology.orchestra.run.smithi094.stdout:skipping the directory /dev 2023-11-24T06:06:29.829 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:06:29.974 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-11-24T06:06:30.055 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-11-24T06:06:30.150 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-volume-2:18.0.0-7484.ga13ea5ff.el8.noarch 97/134 2023-11-24T06:06:30.298 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-11-24T06:06:30.311 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-11-24T06:06:30.465 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-11-24T06:06:30.474 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-11-24T06:06:30.609 INFO:teuthology.orchestra.run.smithi091.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-11-24T06:06:30.609 INFO:teuthology.orchestra.run.smithi091.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-24T06:06:30.851 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-24T06:06:30.851 INFO:teuthology.orchestra.run.smithi091.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-24T06:06:31.095 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-24T06:06:31.305 INFO:teuthology.orchestra.run.smithi091.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-24T06:06:31.749 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-volume-2:18.0.0-7484.ga13ea5ff.el8.noarch 97/134 2023-11-24T06:06:31.810 INFO:teuthology.orchestra.run.smithi091.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-24T06:06:31.810 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-2:18.0.0-7484.ga13ea5ff.el8.x86_64 1/134 2023-11-24T06:06:31.810 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-base-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2/134 2023-11-24T06:06:31.810 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 3/134 2023-11-24T06:06:31.810 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 4/134 2023-11-24T06:06:31.810 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 5/134 2023-11-24T06:06:31.810 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-mds-2:18.0.0-7484.ga13ea5ff.el8.x86_64 6/134 2023-11-24T06:06:31.810 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-mgr-2:18.0.0-7484.ga13ea5ff.el8.x86_64 7/134 2023-11-24T06:06:31.811 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-mon-2:18.0.0-7484.ga13ea5ff.el8.x86_64 8/134 2023-11-24T06:06:31.811 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-osd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 9/134 2023-11-24T06:06:31.811 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 10/134 2023-11-24T06:06:31.811 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-selinux-2:18.0.0-7484.ga13ea5ff.el8.x86_64 11/134 2023-11-24T06:06:31.811 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-test-2:18.0.0-7484.ga13ea5ff.el8.x86_64 12/134 2023-11-24T06:06:31.811 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 13/134 2023-11-24T06:06:31.811 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_ 14/134 2023-11-24T06:06:31.811 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libcephfs2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 15/134 2023-11-24T06:06:31.811 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libcephsqlite-2:18.0.0-7484.ga13ea5ff.el8.x86_64 16/134 2023-11-24T06:06:31.811 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : librados-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_6 17/134 2023-11-24T06:06:31.811 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libradosstriper1-2:18.0.0-7484.ga13ea5ff.el8.x86 18/134 2023-11-24T06:06:31.811 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : librgw2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 19/134 2023-11-24T06:06:31.812 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el 20/134 2023-11-24T06:06:31.812 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el8. 21/134 2023-11-24T06:06:31.812 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-cephfs-2:18.0.0-7484.ga13ea5ff.el8.x86_6 22/134 2023-11-24T06:06:31.812 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-rados-2:18.0.0-7484.ga13ea5ff.el8.x86_64 23/134 2023-11-24T06:06:31.812 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-rbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 24/134 2023-11-24T06:06:31.812 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-rgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 25/134 2023-11-24T06:06:31.812 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : rbd-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 26/134 2023-11-24T06:06:31.812 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : rbd-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 27/134 2023-11-24T06:06:31.812 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : rbd-nbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 28/134 2023-11-24T06:06:31.812 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff. 29/134 2023-11-24T06:06:31.812 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el8.noa 30/134 2023-11-24T06:06:31.812 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el8.n 31/134 2023-11-24T06:06:31.813 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 32/134 2023-11-24T06:06:31.813 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el 33/134 2023-11-24T06:06:31.813 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el8.noarch 34/134 2023-11-24T06:06:31.813 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.e 35/134 2023-11-24T06:06:31.813 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : ceph-volume-2:18.0.0-7484.ga13ea5ff.el8.noarch 36/134 2023-11-24T06:06:31.813 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 37/134 2023-11-24T06:06:31.813 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : cephfs-top-2:18.0.0-7484.ga13ea5ff.el8.noarch 38/134 2023-11-24T06:06:31.813 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-24T06:06:31.813 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-24T06:06:31.813 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-24T06:06:31.813 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-24T06:06:31.813 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-24T06:06:31.814 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-24T06:06:31.814 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-24T06:06:31.814 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-24T06:06:31.814 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-24T06:06:31.814 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-24T06:06:31.814 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-24T06:06:31.814 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-24T06:06:31.814 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-24T06:06:31.814 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-24T06:06:31.814 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-24T06:06:31.814 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-24T06:06:31.815 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-24T06:06:31.815 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-24T06:06:31.815 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-24T06:06:31.815 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-24T06:06:31.815 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-24T06:06:31.815 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-24T06:06:31.815 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-24T06:06:31.815 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-24T06:06:31.815 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-24T06:06:31.815 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-24T06:06:31.815 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-24T06:06:31.815 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-24T06:06:31.816 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-24T06:06:31.816 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-24T06:06:31.816 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-24T06:06:31.816 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-24T06:06:31.816 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-24T06:06:31.816 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-11-24T06:06:31.816 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-11-24T06:06:31.816 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-11-24T06:06:31.816 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-11-24T06:06:31.816 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-11-24T06:06:31.816 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-11-24T06:06:31.817 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-11-24T06:06:31.817 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-11-24T06:06:31.817 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-11-24T06:06:31.817 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-11-24T06:06:31.817 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-11-24T06:06:31.817 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-11-24T06:06:31.817 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-11-24T06:06:31.817 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-11-24T06:06:31.817 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-11-24T06:06:31.817 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-11-24T06:06:31.817 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-11-24T06:06:31.818 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-11-24T06:06:31.818 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-11-24T06:06:31.818 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-11-24T06:06:31.818 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-11-24T06:06:31.818 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-11-24T06:06:31.818 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-11-24T06:06:31.818 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-11-24T06:06:31.818 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-11-24T06:06:31.818 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-11-24T06:06:31.818 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-11-24T06:06:31.818 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-11-24T06:06:31.818 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-11-24T06:06:31.818 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-11-24T06:06:31.819 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-11-24T06:06:31.819 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-11-24T06:06:31.819 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-11-24T06:06:31.819 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-11-24T06:06:31.819 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-11-24T06:06:31.819 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-11-24T06:06:31.819 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-11-24T06:06:31.819 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-11-24T06:06:31.819 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-11-24T06:06:31.819 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-11-24T06:06:31.819 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-11-24T06:06:31.819 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-11-24T06:06:31.820 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-11-24T06:06:31.820 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-11-24T06:06:31.820 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-11-24T06:06:31.820 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-11-24T06:06:31.820 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-11-24T06:06:31.820 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-11-24T06:06:31.820 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-11-24T06:06:31.820 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-11-24T06:06:31.820 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-11-24T06:06:31.820 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-11-24T06:06:31.820 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-11-24T06:06:31.820 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-11-24T06:06:31.820 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-11-24T06:06:31.821 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-11-24T06:06:31.821 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-11-24T06:06:31.821 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : librados2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 129/134 2023-11-24T06:06:31.821 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-24T06:06:31.821 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : librbd1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 131/134 2023-11-24T06:06:31.821 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-24T06:06:31.821 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-24T06:06:31.960 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-osd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 98/134 2023-11-24T06:06:32.191 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-osd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 98/134 2023-11-24T06:06:32.351 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mds-2:18.0.0-7484.ga13ea5ff.el8.x86_64 99/134 2023-11-24T06:06:32.805 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mds-2:18.0.0-7484.ga13ea5ff.el8.x86_64 99/134 2023-11-24T06:06:33.025 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mon-2:18.0.0-7484.ga13ea5ff.el8.x86_64 100/134 2023-11-24T06:06:33.042 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mon-2:18.0.0-7484.ga13ea5ff.el8.x86_64 100/134 2023-11-24T06:06:34.283 INFO:teuthology.orchestra.run.smithi091.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-24T06:06:34.283 INFO:teuthology.orchestra.run.smithi091.stdout:Last metadata expiration check: 0:01:49 ago on Fri 24 Nov 2023 06:04:45 AM UTC. 2023-11-24T06:06:34.716 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 101/134 2023-11-24T06:06:34.913 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 101/134 2023-11-24T06:06:35.065 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.e 102/134 2023-11-24T06:06:35.267 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff. 103/134 2023-11-24T06:06:35.449 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el8.noa 104/134 2023-11-24T06:06:35.669 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el8.noa 104/134 2023-11-24T06:06:35.984 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el8.n 105/134 2023-11-24T06:06:36.873 INFO:teuthology.orchestra.run.smithi091.stdout:Installed products updated. 2023-11-24T06:06:36.941 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el8.n 105/134 2023-11-24T06:06:37.113 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:06:37.114 INFO:teuthology.orchestra.run.smithi091.stdout:Upgraded: 2023-11-24T06:06:37.114 INFO:teuthology.orchestra.run.smithi091.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-24T06:06:37.114 INFO:teuthology.orchestra.run.smithi091.stdout: librados2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.114 INFO:teuthology.orchestra.run.smithi091.stdout: librbd1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.114 INFO:teuthology.orchestra.run.smithi091.stdout:Installed: 2023-11-24T06:06:37.114 INFO:teuthology.orchestra.run.smithi091.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-24T06:06:37.114 INFO:teuthology.orchestra.run.smithi091.stdout: automake-1.16.1-7.el8.noarch 2023-11-24T06:06:37.114 INFO:teuthology.orchestra.run.smithi091.stdout: bison-3.0.4-10.el8.x86_64 2023-11-24T06:06:37.114 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.114 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-base-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.114 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.115 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.115 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:37.115 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-immutable-object-cache-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.115 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mds-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.115 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.115 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:37.115 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:37.115 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:37.115 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:37.115 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:37.115 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-mon-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.115 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-osd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.116 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:37.116 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.116 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-selinux-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.116 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-test-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.116 INFO:teuthology.orchestra.run.smithi091.stdout: ceph-volume-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:37.116 INFO:teuthology.orchestra.run.smithi091.stdout: cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:37.116 INFO:teuthology.orchestra.run.smithi091.stdout: cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.116 INFO:teuthology.orchestra.run.smithi091.stdout: cephfs-top-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:37.116 INFO:teuthology.orchestra.run.smithi091.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-24T06:06:37.116 INFO:teuthology.orchestra.run.smithi091.stdout: flex-2.6.1-9.el8.x86_64 2023-11-24T06:06:37.116 INFO:teuthology.orchestra.run.smithi091.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-24T06:06:37.116 INFO:teuthology.orchestra.run.smithi091.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-24T06:06:37.117 INFO:teuthology.orchestra.run.smithi091.stdout: jq-1.6-3.el8.x86_64 2023-11-24T06:06:37.117 INFO:teuthology.orchestra.run.smithi091.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-24T06:06:37.117 INFO:teuthology.orchestra.run.smithi091.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-24T06:06:37.117 INFO:teuthology.orchestra.run.smithi091.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-24T06:06:37.117 INFO:teuthology.orchestra.run.smithi091.stdout: libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.117 INFO:teuthology.orchestra.run.smithi091.stdout: libcephfs2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.117 INFO:teuthology.orchestra.run.smithi091.stdout: libcephsqlite-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.117 INFO:teuthology.orchestra.run.smithi091.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-24T06:06:37.117 INFO:teuthology.orchestra.run.smithi091.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-24T06:06:37.117 INFO:teuthology.orchestra.run.smithi091.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-24T06:06:37.117 INFO:teuthology.orchestra.run.smithi091.stdout: libpq-13.5-1.el8.x86_64 2023-11-24T06:06:37.117 INFO:teuthology.orchestra.run.smithi091.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-24T06:06:37.118 INFO:teuthology.orchestra.run.smithi091.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-24T06:06:37.118 INFO:teuthology.orchestra.run.smithi091.stdout: librados-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.118 INFO:teuthology.orchestra.run.smithi091.stdout: libradosstriper1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.118 INFO:teuthology.orchestra.run.smithi091.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-24T06:06:37.118 INFO:teuthology.orchestra.run.smithi091.stdout: librgw2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.118 INFO:teuthology.orchestra.run.smithi091.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-24T06:06:37.118 INFO:teuthology.orchestra.run.smithi091.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-24T06:06:37.118 INFO:teuthology.orchestra.run.smithi091.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-24T06:06:37.118 INFO:teuthology.orchestra.run.smithi091.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-24T06:06:37.118 INFO:teuthology.orchestra.run.smithi091.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-24T06:06:37.118 INFO:teuthology.orchestra.run.smithi091.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-24T06:06:37.118 INFO:teuthology.orchestra.run.smithi091.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-24T06:06:37.119 INFO:teuthology.orchestra.run.smithi091.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-24T06:06:37.119 INFO:teuthology.orchestra.run.smithi091.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-24T06:06:37.119 INFO:teuthology.orchestra.run.smithi091.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-24T06:06:37.119 INFO:teuthology.orchestra.run.smithi091.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-24T06:06:37.119 INFO:teuthology.orchestra.run.smithi091.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-24T06:06:37.119 INFO:teuthology.orchestra.run.smithi091.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-24T06:06:37.119 INFO:teuthology.orchestra.run.smithi091.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-24T06:06:37.119 INFO:teuthology.orchestra.run.smithi091.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-24T06:06:37.119 INFO:teuthology.orchestra.run.smithi091.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-24T06:06:37.119 INFO:teuthology.orchestra.run.smithi091.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-24T06:06:37.119 INFO:teuthology.orchestra.run.smithi091.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-24T06:06:37.119 INFO:teuthology.orchestra.run.smithi091.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-24T06:06:37.120 INFO:teuthology.orchestra.run.smithi091.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-24T06:06:37.120 INFO:teuthology.orchestra.run.smithi091.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-24T06:06:37.120 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-24T06:06:37.120 INFO:teuthology.orchestra.run.smithi091.stdout: python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.120 INFO:teuthology.orchestra.run.smithi091.stdout: python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.120 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cephfs-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.120 INFO:teuthology.orchestra.run.smithi091.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-24T06:06:37.120 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-24T06:06:37.120 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-24T06:06:37.120 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-24T06:06:37.120 INFO:teuthology.orchestra.run.smithi091.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-24T06:06:37.120 INFO:teuthology.orchestra.run.smithi091.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-24T06:06:37.121 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-24T06:06:37.121 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-24T06:06:37.121 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-24T06:06:37.121 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-24T06:06:37.121 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-24T06:06:37.121 INFO:teuthology.orchestra.run.smithi091.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-24T06:06:37.121 INFO:teuthology.orchestra.run.smithi091.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-24T06:06:37.121 INFO:teuthology.orchestra.run.smithi091.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-24T06:06:37.121 INFO:teuthology.orchestra.run.smithi091.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-24T06:06:37.121 INFO:teuthology.orchestra.run.smithi091.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-24T06:06:37.121 INFO:teuthology.orchestra.run.smithi091.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-24T06:06:37.121 INFO:teuthology.orchestra.run.smithi091.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-24T06:06:37.121 INFO:teuthology.orchestra.run.smithi091.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-24T06:06:37.122 INFO:teuthology.orchestra.run.smithi091.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-24T06:06:37.122 INFO:teuthology.orchestra.run.smithi091.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-24T06:06:37.122 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-24T06:06:37.122 INFO:teuthology.orchestra.run.smithi091.stdout: python3-portend-2.6-1.el8.noarch 2023-11-24T06:06:37.122 INFO:teuthology.orchestra.run.smithi091.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-24T06:06:37.122 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-24T06:06:37.122 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-24T06:06:37.122 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-24T06:06:37.122 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-24T06:06:37.122 INFO:teuthology.orchestra.run.smithi091.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-24T06:06:37.122 INFO:teuthology.orchestra.run.smithi091.stdout: python3-rados-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.123 INFO:teuthology.orchestra.run.smithi091.stdout: python3-rbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.123 INFO:teuthology.orchestra.run.smithi091.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-24T06:06:37.123 INFO:teuthology.orchestra.run.smithi091.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-24T06:06:37.123 INFO:teuthology.orchestra.run.smithi091.stdout: python3-rgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.123 INFO:teuthology.orchestra.run.smithi091.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-24T06:06:37.123 INFO:teuthology.orchestra.run.smithi091.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-24T06:06:37.123 INFO:teuthology.orchestra.run.smithi091.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-24T06:06:37.123 INFO:teuthology.orchestra.run.smithi091.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-24T06:06:37.123 INFO:teuthology.orchestra.run.smithi091.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-24T06:06:37.123 INFO:teuthology.orchestra.run.smithi091.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-24T06:06:37.123 INFO:teuthology.orchestra.run.smithi091.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-24T06:06:37.123 INFO:teuthology.orchestra.run.smithi091.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-24T06:06:37.123 INFO:teuthology.orchestra.run.smithi091.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-24T06:06:37.124 INFO:teuthology.orchestra.run.smithi091.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-24T06:06:37.124 INFO:teuthology.orchestra.run.smithi091.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-24T06:06:37.124 INFO:teuthology.orchestra.run.smithi091.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-24T06:06:37.124 INFO:teuthology.orchestra.run.smithi091.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-24T06:06:37.124 INFO:teuthology.orchestra.run.smithi091.stdout: rbd-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.124 INFO:teuthology.orchestra.run.smithi091.stdout: rbd-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.124 INFO:teuthology.orchestra.run.smithi091.stdout: rbd-nbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:37.124 INFO:teuthology.orchestra.run.smithi091.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-24T06:06:37.124 INFO:teuthology.orchestra.run.smithi091.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-24T06:06:37.124 INFO:teuthology.orchestra.run.smithi091.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-24T06:06:37.124 INFO:teuthology.orchestra.run.smithi091.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-24T06:06:37.124 INFO:teuthology.orchestra.run.smithi091.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-24T06:06:37.125 INFO:teuthology.orchestra.run.smithi091.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-24T06:06:37.125 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:06:37.125 INFO:teuthology.orchestra.run.smithi091.stdout:Complete! 2023-11-24T06:06:37.141 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 106/134 2023-11-24T06:06:37.232 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 106/134 2023-11-24T06:06:37.327 DEBUG:teuthology.parallel:result is None 2023-11-24T06:06:37.593 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el 107/134 2023-11-24T06:06:37.768 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-2:18.0.0-7484.ga13ea5ff.el8.x86_64 108/134 2023-11-24T06:06:37.805 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7484.ga13ea5ff.el8.x86_64 108/134 2023-11-24T06:06:38.016 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el8.noarch 109/134 2023-11-24T06:06:38.027 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el8.noarch 109/134 2023-11-24T06:06:39.346 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-2:18.0.0-7484.ga13ea5ff.el8.x86_64 110/134 2023-11-24T06:06:39.544 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 111/134 2023-11-24T06:06:39.579 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 111/134 2023-11-24T06:06:39.733 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 112/134 2023-11-24T06:06:39.775 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 112/134 2023-11-24T06:06:39.908 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 113/134 2023-11-24T06:06:40.266 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 113/134 2023-11-24T06:06:40.424 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 114/134 2023-11-24T06:06:45.094 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 114/134 2023-11-24T06:06:45.418 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-test-2:18.0.0-7484.ga13ea5ff.el8.x86_64 115/134 2023-11-24T06:06:45.573 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-11-24T06:06:45.765 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-11-24T06:06:46.646 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-11-24T06:06:46.943 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-11-24T06:06:47.107 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : ceph-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 119/134 2023-11-24T06:06:47.302 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-24T06:06:47.486 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_ 121/134 2023-11-24T06:06:47.633 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-24T06:06:49.793 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-24T06:06:50.032 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-24T06:06:50.054 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-11-24T06:06:50.211 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : cephfs-top-2:18.0.0-7484.ga13ea5ff.el8.noarch 124/134 2023-11-24T06:06:50.528 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 125/134 2023-11-24T06:06:50.804 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : rbd-nbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 126/134 2023-11-24T06:06:51.150 INFO:teuthology.orchestra.run.smithi094.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-11-24T06:06:51.326 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-24T06:06:51.337 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-24T06:06:51.530 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-11-24T06:06:51.698 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-11-24T06:06:51.782 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-11-24T06:06:51.997 INFO:teuthology.orchestra.run.smithi094.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-11-24T06:06:52.010 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-11-24T06:06:52.010 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-24T06:06:52.209 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-24T06:06:52.210 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-24T06:06:52.461 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-11-24T06:06:52.629 INFO:teuthology.orchestra.run.smithi094.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-24T06:06:53.143 INFO:teuthology.orchestra.run.smithi094.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-11-24T06:06:53.143 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-2:18.0.0-7484.ga13ea5ff.el8.x86_64 1/134 2023-11-24T06:06:53.143 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-base-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2/134 2023-11-24T06:06:53.143 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 3/134 2023-11-24T06:06:53.143 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 4/134 2023-11-24T06:06:53.144 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7484.ga13ea 5/134 2023-11-24T06:06:53.144 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mds-2:18.0.0-7484.ga13ea5ff.el8.x86_64 6/134 2023-11-24T06:06:53.144 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-2:18.0.0-7484.ga13ea5ff.el8.x86_64 7/134 2023-11-24T06:06:53.144 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mon-2:18.0.0-7484.ga13ea5ff.el8.x86_64 8/134 2023-11-24T06:06:53.144 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-osd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 9/134 2023-11-24T06:06:53.144 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 10/134 2023-11-24T06:06:53.144 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-selinux-2:18.0.0-7484.ga13ea5ff.el8.x86_64 11/134 2023-11-24T06:06:53.144 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-test-2:18.0.0-7484.ga13ea5ff.el8.x86_64 12/134 2023-11-24T06:06:53.144 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 13/134 2023-11-24T06:06:53.144 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_ 14/134 2023-11-24T06:06:53.144 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephfs2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 15/134 2023-11-24T06:06:53.144 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcephsqlite-2:18.0.0-7484.ga13ea5ff.el8.x86_64 16/134 2023-11-24T06:06:53.145 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_6 17/134 2023-11-24T06:06:53.145 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libradosstriper1-2:18.0.0-7484.ga13ea5ff.el8.x86 18/134 2023-11-24T06:06:53.145 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librgw2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 19/134 2023-11-24T06:06:53.145 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el 20/134 2023-11-24T06:06:53.145 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el8. 21/134 2023-11-24T06:06:53.145 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cephfs-2:18.0.0-7484.ga13ea5ff.el8.x86_6 22/134 2023-11-24T06:06:53.145 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rados-2:18.0.0-7484.ga13ea5ff.el8.x86_64 23/134 2023-11-24T06:06:53.145 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 24/134 2023-11-24T06:06:53.145 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 25/134 2023-11-24T06:06:53.145 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 26/134 2023-11-24T06:06:53.145 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 27/134 2023-11-24T06:06:53.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : rbd-nbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 28/134 2023-11-24T06:06:53.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff. 29/134 2023-11-24T06:06:53.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el8.noa 30/134 2023-11-24T06:06:53.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el8.n 31/134 2023-11-24T06:06:53.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13 32/134 2023-11-24T06:06:53.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el 33/134 2023-11-24T06:06:53.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el8.noarch 34/134 2023-11-24T06:06:53.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.e 35/134 2023-11-24T06:06:53.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : ceph-volume-2:18.0.0-7484.ga13ea5ff.el8.noarch 36/134 2023-11-24T06:06:53.155 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 37/134 2023-11-24T06:06:53.156 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : cephfs-top-2:18.0.0-7484.ga13ea5ff.el8.noarch 38/134 2023-11-24T06:06:53.156 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-11-24T06:06:53.156 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-11-24T06:06:53.156 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-11-24T06:06:53.156 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-11-24T06:06:53.156 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-11-24T06:06:53.156 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-11-24T06:06:53.156 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-11-24T06:06:53.156 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-11-24T06:06:53.156 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-11-24T06:06:53.156 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-11-24T06:06:53.156 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-11-24T06:06:53.157 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-11-24T06:06:53.157 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-11-24T06:06:53.157 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-11-24T06:06:53.157 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-11-24T06:06:53.157 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-11-24T06:06:53.157 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-11-24T06:06:53.157 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-11-24T06:06:53.157 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-11-24T06:06:53.157 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-11-24T06:06:53.157 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-11-24T06:06:53.158 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-11-24T06:06:53.158 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-11-24T06:06:53.158 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-11-24T06:06:53.158 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-11-24T06:06:53.158 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-11-24T06:06:53.158 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-11-24T06:06:53.158 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-11-24T06:06:53.158 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-11-24T06:06:53.158 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-11-24T06:06:53.158 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-11-24T06:06:53.158 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-11-24T06:06:53.159 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-11-24T06:06:53.159 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-11-24T06:06:53.159 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-11-24T06:06:53.159 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-11-24T06:06:53.159 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-11-24T06:06:53.159 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-11-24T06:06:53.159 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-11-24T06:06:53.159 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-11-24T06:06:53.159 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-11-24T06:06:53.159 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-11-24T06:06:53.159 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-11-24T06:06:53.159 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-11-24T06:06:53.160 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-11-24T06:06:53.160 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-11-24T06:06:53.160 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-11-24T06:06:53.160 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-11-24T06:06:53.160 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-11-24T06:06:53.160 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-11-24T06:06:53.160 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-11-24T06:06:53.160 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-11-24T06:06:53.160 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-11-24T06:06:53.160 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-11-24T06:06:53.160 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-11-24T06:06:53.160 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-11-24T06:06:53.161 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-11-24T06:06:53.161 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-11-24T06:06:53.161 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-11-24T06:06:53.161 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-11-24T06:06:53.161 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-11-24T06:06:53.161 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-11-24T06:06:53.161 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-11-24T06:06:53.161 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-11-24T06:06:53.161 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-11-24T06:06:53.161 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-11-24T06:06:53.162 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-11-24T06:06:53.162 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-11-24T06:06:53.162 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-11-24T06:06:53.162 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-11-24T06:06:53.162 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-11-24T06:06:53.162 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-11-24T06:06:53.162 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-11-24T06:06:53.162 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-11-24T06:06:53.162 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-11-24T06:06:53.162 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-11-24T06:06:53.162 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-11-24T06:06:53.163 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-11-24T06:06:53.163 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-11-24T06:06:53.163 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-11-24T06:06:53.163 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-11-24T06:06:53.163 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-11-24T06:06:53.163 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-11-24T06:06:53.163 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-11-24T06:06:53.163 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-11-24T06:06:53.163 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-11-24T06:06:53.163 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-11-24T06:06:53.163 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-11-24T06:06:53.163 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-11-24T06:06:53.164 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-11-24T06:06:53.164 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 129/134 2023-11-24T06:06:53.164 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-11-24T06:06:53.164 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 131/134 2023-11-24T06:06:53.164 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-11-24T06:06:53.164 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-11-24T06:06:55.698 INFO:teuthology.orchestra.run.smithi094.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-11-24T06:06:55.699 INFO:teuthology.orchestra.run.smithi094.stdout:Last metadata expiration check: 0:01:54 ago on Fri 24 Nov 2023 06:05:01 AM UTC. 2023-11-24T06:06:58.726 INFO:teuthology.orchestra.run.smithi094.stdout:Installed products updated. 2023-11-24T06:06:58.951 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:06:58.952 INFO:teuthology.orchestra.run.smithi094.stdout:Upgraded: 2023-11-24T06:06:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-11-24T06:06:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: librados2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: librbd1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.952 INFO:teuthology.orchestra.run.smithi094.stdout:Installed: 2023-11-24T06:06:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-11-24T06:06:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: automake-1.16.1-7.el8.noarch 2023-11-24T06:06:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: bison-3.0.4-10.el8.x86_64 2023-11-24T06:06:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-base-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.952 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-grafana-dashboards-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-immutable-object-cache-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mds-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-dashboard-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-modules-core-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mgr-rook-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-mon-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-osd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-prometheus-alerts-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:58.953 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-radosgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-selinux-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-test-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: ceph-volume-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: cephadm-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: cephfs-top-2:18.0.0-7484.ga13ea5ff.el8.noarch 2023-11-24T06:06:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-11-24T06:06:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: flex-2.6.1-9.el8.x86_64 2023-11-24T06:06:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: fuse-2.9.7-15.el8.x86_64 2023-11-24T06:06:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-24T06:06:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: jq-1.6-3.el8.x86_64 2023-11-24T06:06:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-11-24T06:06:58.954 INFO:teuthology.orchestra.run.smithi094.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-11-24T06:06:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-11-24T06:06:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: libcephfs2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: libcephsqlite-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-11-24T06:06:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-24T06:06:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-24T06:06:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: libpq-13.5-1.el8.x86_64 2023-11-24T06:06:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-24T06:06:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-24T06:06:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: librados-devel-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: libradosstriper1-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.955 INFO:teuthology.orchestra.run.smithi094.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-24T06:06:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: librgw2-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-11-24T06:06:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-11-24T06:06:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: libtool-2.4.6-25.el8.x86_64 2023-11-24T06:06:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-24T06:06:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-11-24T06:06:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-11-24T06:06:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-24T06:06:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-24T06:06:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-24T06:06:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-24T06:06:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-11-24T06:06:58.956 INFO:teuthology.orchestra.run.smithi094.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-11-24T06:06:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-11-24T06:06:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-11-24T06:06:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-24T06:06:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-24T06:06:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-11-24T06:06:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-24T06:06:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-24T06:06:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-24T06:06:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-24T06:06:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-24T06:06:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-argparse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: python3-ceph-common-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.957 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cephfs-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-24T06:06:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-24T06:06:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-24T06:06:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-24T06:06:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-24T06:06:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-24T06:06:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-24T06:06:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-24T06:06:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-24T06:06:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-24T06:06:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-24T06:06:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-24T06:06:58.958 INFO:teuthology.orchestra.run.smithi094.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-24T06:06:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-24T06:06:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-24T06:06:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-24T06:06:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-24T06:06:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-24T06:06:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-24T06:06:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-24T06:06:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-24T06:06:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-24T06:06:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: python3-portend-2.6-1.el8.noarch 2023-11-24T06:06:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-24T06:06:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-24T06:06:58.959 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-24T06:06:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-24T06:06:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-24T06:06:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-24T06:06:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rados-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-24T06:06:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-24T06:06:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rgw-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-24T06:06:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-24T06:06:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-24T06:06:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-24T06:06:58.960 INFO:teuthology.orchestra.run.smithi094.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-24T06:06:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-24T06:06:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-24T06:06:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-24T06:06:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-24T06:06:58.961 INFO:teuthology.orchestra.run.smithi094.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-24T06:06:58.962 INFO:teuthology.orchestra.run.smithi094.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-24T06:06:58.962 INFO:teuthology.orchestra.run.smithi094.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-24T06:06:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-24T06:06:58.963 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-fuse-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.964 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-mirror-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.964 INFO:teuthology.orchestra.run.smithi094.stdout: rbd-nbd-2:18.0.0-7484.ga13ea5ff.el8.x86_64 2023-11-24T06:06:58.965 INFO:teuthology.orchestra.run.smithi094.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-24T06:06:58.965 INFO:teuthology.orchestra.run.smithi094.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-24T06:06:58.965 INFO:teuthology.orchestra.run.smithi094.stdout: uuid-1.6.2-43.el8.x86_64 2023-11-24T06:06:58.966 INFO:teuthology.orchestra.run.smithi094.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-11-24T06:06:58.967 INFO:teuthology.orchestra.run.smithi094.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-24T06:06:58.968 INFO:teuthology.orchestra.run.smithi094.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-11-24T06:06:58.968 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:06:58.968 INFO:teuthology.orchestra.run.smithi094.stdout:Complete! 2023-11-24T06:06:59.143 DEBUG:teuthology.parallel:result is None 2023-11-24T06:06:59.144 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a13ea5ff00cb4df7a9cc941166ddeb358881144c 2023-11-24T06:06:59.519 DEBUG:teuthology.orchestra.run.smithi091:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-24T06:06:59.543 INFO:teuthology.orchestra.run.smithi091.stdout:18.0.0-7484.ga13ea5ff.el8 2023-11-24T06:06:59.544 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7484.ga13ea5ff.el8 2023-11-24T06:06:59.544 INFO:teuthology.task.install:The correct ceph version 18.0.0-7484.ga13ea5ff is installed. 2023-11-24T06:06:59.545 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a13ea5ff00cb4df7a9cc941166ddeb358881144c 2023-11-24T06:06:59.896 DEBUG:teuthology.orchestra.run.smithi094:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-24T06:06:59.919 INFO:teuthology.orchestra.run.smithi094.stdout:18.0.0-7484.ga13ea5ff.el8 2023-11-24T06:06:59.919 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7484.ga13ea5ff.el8 2023-11-24T06:06:59.919 INFO:teuthology.task.install:The correct ceph version 18.0.0-7484.ga13ea5ff is installed. 2023-11-24T06:06:59.919 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-24T06:06:59.919 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:06:59.920 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-24T06:06:59.968 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:06:59.968 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-24T06:07:00.010 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-24T06:07:00.010 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:00.010 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd of=/usr/bin/daemon-helper 2023-11-24T06:07:00.041 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-24T06:07:00.112 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:00.113 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/daemon-helper 2023-11-24T06:07:00.139 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-24T06:07:00.206 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-24T06:07:00.206 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:00.206 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-24T06:07:00.236 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-24T06:07:00.305 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:00.305 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-24T06:07:00.330 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-24T06:07:00.396 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-24T06:07:00.397 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:00.397 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd of=/usr/bin/stdin-killer 2023-11-24T06:07:00.425 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-24T06:07:00.494 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:00.494 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/usr/bin/stdin-killer 2023-11-24T06:07:00.521 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-24T06:07:00.587 INFO:teuthology.run_tasks:Running task ceph... 2023-11-24T06:07:00.664 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-24T06:07:00.664 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod 777 /var/log/ceph 2023-11-24T06:07:00.667 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 777 /var/log/ceph 2023-11-24T06:07:00.693 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-24T06:07:00.694 DEBUG:teuthology.orchestra.run.smithi091:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-24T06:07:00.736 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-24T06:07:00.762 INFO:tasks.ceph:Creating extra log directories... 2023-11-24T06:07:00.762 DEBUG:teuthology.orchestra.run.smithi091:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-24T06:07:00.804 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-24T06:07:00.836 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-11-24T06:07:00.837 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-11-24T06:07:00.837 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-11-24T06:07:00.837 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:00.837 DEBUG:teuthology.orchestra.run.smithi091:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-24T06:07:00.888 DEBUG:teuthology.orchestra.run.smithi091:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-24T06:07:01.023 DEBUG:teuthology.orchestra.run.smithi091:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-11-24T06:07:01.049 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:01.049 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-11-24T06:07:01.065 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-11-24T06:07:01.195 DEBUG:teuthology.orchestra.run.smithi094:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-11-24T06:07:01.228 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-24T06:07:01.228 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-24T06:07:01.228 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547', 'branch': 'rishabh-mgr-vol-misc', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}}', 'email': None, 'first_in_suite': False, 'job_id': '7465547', 'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'rishabh-mgr-vol-misc'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-mgr-vol-misc', 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 3305, 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c', 'sleep_before_teardown': 0, 'suite': 'fs:volumes', 'suite_branch': 'rishabh-mgr-vol-misc', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi091.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs=', 'smithi094.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '8247bbb5b470f7e2a1f3eefcc16e24cf1338f606'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'a13ea5ff00cb4df7a9cc941166ddeb358881144c', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '2442239d2653456406c25ae0c71b689c8f2657b6', 'timestamp': '2023-11-24_05:41:33', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-11-24T06:07:01.229 DEBUG:teuthology.orchestra.run.smithi091:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-24T06:07:01.231 DEBUG:teuthology.orchestra.run.smithi094:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-24T06:07:01.279 DEBUG:teuthology.orchestra.run.smithi091:> sudo install -d -m0777 -- /var/run/ceph 2023-11-24T06:07:01.288 DEBUG:teuthology.orchestra.run.smithi094:> sudo install -d -m0777 -- /var/run/ceph 2023-11-24T06:07:01.348 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:01.348 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/scratch_devs of=/dev/stdout 2023-11-24T06:07:01.368 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-24T06:07:01.369 DEBUG:teuthology.orchestra.run.smithi091:> stat /dev/vg_nvme/lv_1 2023-11-24T06:07:01.425 INFO:teuthology.orchestra.run.smithi091.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-24T06:07:01.425 INFO:teuthology.orchestra.run.smithi091.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T06:07:01.425 INFO:teuthology.orchestra.run.smithi091.stdout:Device: 6h/6d Inode: 393232 Links: 1 2023-11-24T06:07:01.425 INFO:teuthology.orchestra.run.smithi091.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T06:07:01.425 INFO:teuthology.orchestra.run.smithi091.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T06:07:01.425 INFO:teuthology.orchestra.run.smithi091.stdout:Access: 2023-11-24 06:06:31.500300181 +0000 2023-11-24T06:07:01.425 INFO:teuthology.orchestra.run.smithi091.stdout:Modify: 2023-11-24 06:03:42.299101937 +0000 2023-11-24T06:07:01.425 INFO:teuthology.orchestra.run.smithi091.stdout:Change: 2023-11-24 06:03:42.299101937 +0000 2023-11-24T06:07:01.426 INFO:teuthology.orchestra.run.smithi091.stdout: Birth: - 2023-11-24T06:07:01.426 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-24T06:07:01.489 INFO:teuthology.orchestra.run.smithi091.stderr:1+0 records in 2023-11-24T06:07:01.490 INFO:teuthology.orchestra.run.smithi091.stderr:1+0 records out 2023-11-24T06:07:01.490 INFO:teuthology.orchestra.run.smithi091.stderr:512 bytes copied, 0.000263824 s, 1.9 MB/s 2023-11-24T06:07:01.491 DEBUG:teuthology.orchestra.run.smithi091:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-24T06:07:01.550 DEBUG:teuthology.orchestra.run.smithi091:> stat /dev/vg_nvme/lv_2 2023-11-24T06:07:01.606 INFO:teuthology.orchestra.run.smithi091.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-24T06:07:01.606 INFO:teuthology.orchestra.run.smithi091.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T06:07:01.606 INFO:teuthology.orchestra.run.smithi091.stdout:Device: 6h/6d Inode: 391611 Links: 1 2023-11-24T06:07:01.606 INFO:teuthology.orchestra.run.smithi091.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T06:07:01.606 INFO:teuthology.orchestra.run.smithi091.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T06:07:01.606 INFO:teuthology.orchestra.run.smithi091.stdout:Access: 2023-11-24 06:06:31.501300156 +0000 2023-11-24T06:07:01.607 INFO:teuthology.orchestra.run.smithi091.stdout:Modify: 2023-11-24 06:03:42.812088978 +0000 2023-11-24T06:07:01.607 INFO:teuthology.orchestra.run.smithi091.stdout:Change: 2023-11-24 06:03:42.812088978 +0000 2023-11-24T06:07:01.607 INFO:teuthology.orchestra.run.smithi091.stdout: Birth: - 2023-11-24T06:07:01.607 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-24T06:07:01.671 INFO:teuthology.orchestra.run.smithi091.stderr:1+0 records in 2023-11-24T06:07:01.671 INFO:teuthology.orchestra.run.smithi091.stderr:1+0 records out 2023-11-24T06:07:01.671 INFO:teuthology.orchestra.run.smithi091.stderr:512 bytes copied, 0.000246509 s, 2.1 MB/s 2023-11-24T06:07:01.672 DEBUG:teuthology.orchestra.run.smithi091:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-24T06:07:01.729 DEBUG:teuthology.orchestra.run.smithi091:> stat /dev/vg_nvme/lv_3 2023-11-24T06:07:01.786 INFO:teuthology.orchestra.run.smithi091.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-24T06:07:01.786 INFO:teuthology.orchestra.run.smithi091.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T06:07:01.786 INFO:teuthology.orchestra.run.smithi091.stdout:Device: 6h/6d Inode: 387941 Links: 1 2023-11-24T06:07:01.786 INFO:teuthology.orchestra.run.smithi091.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T06:07:01.787 INFO:teuthology.orchestra.run.smithi091.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T06:07:01.787 INFO:teuthology.orchestra.run.smithi091.stdout:Access: 2023-11-24 06:06:31.501300156 +0000 2023-11-24T06:07:01.787 INFO:teuthology.orchestra.run.smithi091.stdout:Modify: 2023-11-24 06:03:43.322076094 +0000 2023-11-24T06:07:01.787 INFO:teuthology.orchestra.run.smithi091.stdout:Change: 2023-11-24 06:03:43.322076094 +0000 2023-11-24T06:07:01.787 INFO:teuthology.orchestra.run.smithi091.stdout: Birth: - 2023-11-24T06:07:01.787 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-24T06:07:01.851 INFO:teuthology.orchestra.run.smithi091.stderr:1+0 records in 2023-11-24T06:07:01.851 INFO:teuthology.orchestra.run.smithi091.stderr:1+0 records out 2023-11-24T06:07:01.851 INFO:teuthology.orchestra.run.smithi091.stderr:512 bytes copied, 0.000220343 s, 2.3 MB/s 2023-11-24T06:07:01.852 DEBUG:teuthology.orchestra.run.smithi091:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-24T06:07:01.910 DEBUG:teuthology.orchestra.run.smithi091:> stat /dev/vg_nvme/lv_4 2023-11-24T06:07:01.967 INFO:teuthology.orchestra.run.smithi091.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-24T06:07:01.967 INFO:teuthology.orchestra.run.smithi091.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T06:07:01.967 INFO:teuthology.orchestra.run.smithi091.stdout:Device: 6h/6d Inode: 391766 Links: 1 2023-11-24T06:07:01.967 INFO:teuthology.orchestra.run.smithi091.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T06:07:01.968 INFO:teuthology.orchestra.run.smithi091.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T06:07:01.968 INFO:teuthology.orchestra.run.smithi091.stdout:Access: 2023-11-24 06:06:31.501300156 +0000 2023-11-24T06:07:01.968 INFO:teuthology.orchestra.run.smithi091.stdout:Modify: 2023-11-24 06:03:43.818063564 +0000 2023-11-24T06:07:01.968 INFO:teuthology.orchestra.run.smithi091.stdout:Change: 2023-11-24 06:03:43.818063564 +0000 2023-11-24T06:07:01.968 INFO:teuthology.orchestra.run.smithi091.stdout: Birth: - 2023-11-24T06:07:01.968 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-24T06:07:02.033 INFO:teuthology.orchestra.run.smithi091.stderr:1+0 records in 2023-11-24T06:07:02.033 INFO:teuthology.orchestra.run.smithi091.stderr:1+0 records out 2023-11-24T06:07:02.033 INFO:teuthology.orchestra.run.smithi091.stderr:512 bytes copied, 0.000185277 s, 2.8 MB/s 2023-11-24T06:07:02.034 DEBUG:teuthology.orchestra.run.smithi091:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-24T06:07:02.091 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-24T06:07:02.092 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:02.092 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/scratch_devs of=/dev/stdout 2023-11-24T06:07:02.108 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-24T06:07:02.108 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_1 2023-11-24T06:07:02.164 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-24T06:07:02.165 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T06:07:02.165 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 381742 Links: 1 2023-11-24T06:07:02.165 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T06:07:02.165 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T06:07:02.165 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2023-11-24 06:06:52.827520322 +0000 2023-11-24T06:07:02.165 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2023-11-24 06:03:55.608865976 +0000 2023-11-24T06:07:02.165 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2023-11-24 06:03:55.608865976 +0000 2023-11-24T06:07:02.165 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2023-11-24T06:07:02.165 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-24T06:07:02.230 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2023-11-24T06:07:02.231 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2023-11-24T06:07:02.231 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.000105845 s, 4.8 MB/s 2023-11-24T06:07:02.232 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-24T06:07:02.289 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_2 2023-11-24T06:07:02.348 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-24T06:07:02.348 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T06:07:02.348 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 380792 Links: 1 2023-11-24T06:07:02.348 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T06:07:02.348 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T06:07:02.348 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2023-11-24 06:06:52.827520322 +0000 2023-11-24T06:07:02.348 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2023-11-24 06:03:56.106857517 +0000 2023-11-24T06:07:02.348 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2023-11-24 06:03:56.106857517 +0000 2023-11-24T06:07:02.349 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2023-11-24T06:07:02.349 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-24T06:07:02.413 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2023-11-24T06:07:02.413 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2023-11-24T06:07:02.413 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 9.1426e-05 s, 5.6 MB/s 2023-11-24T06:07:02.414 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-24T06:07:02.471 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_3 2023-11-24T06:07:02.527 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-24T06:07:02.528 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T06:07:02.528 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 391307 Links: 1 2023-11-24T06:07:02.528 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T06:07:02.528 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T06:07:02.528 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2023-11-24 06:06:52.827520322 +0000 2023-11-24T06:07:02.528 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2023-11-24 06:03:56.606849025 +0000 2023-11-24T06:07:02.528 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2023-11-24 06:03:56.606849025 +0000 2023-11-24T06:07:02.528 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2023-11-24T06:07:02.528 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-24T06:07:02.592 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2023-11-24T06:07:02.592 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2023-11-24T06:07:02.592 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 0.00011482 s, 4.5 MB/s 2023-11-24T06:07:02.593 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-24T06:07:02.652 DEBUG:teuthology.orchestra.run.smithi094:> stat /dev/vg_nvme/lv_4 2023-11-24T06:07:02.708 INFO:teuthology.orchestra.run.smithi094.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-24T06:07:02.708 INFO:teuthology.orchestra.run.smithi094.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-24T06:07:02.708 INFO:teuthology.orchestra.run.smithi094.stdout:Device: 6h/6d Inode: 391400 Links: 1 2023-11-24T06:07:02.708 INFO:teuthology.orchestra.run.smithi094.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-24T06:07:02.708 INFO:teuthology.orchestra.run.smithi094.stdout:Context: system_u:object_r:device_t:s0 2023-11-24T06:07:02.709 INFO:teuthology.orchestra.run.smithi094.stdout:Access: 2023-11-24 06:06:52.827520322 +0000 2023-11-24T06:07:02.709 INFO:teuthology.orchestra.run.smithi094.stdout:Modify: 2023-11-24 06:03:57.100840634 +0000 2023-11-24T06:07:02.709 INFO:teuthology.orchestra.run.smithi094.stdout:Change: 2023-11-24 06:03:57.100840634 +0000 2023-11-24T06:07:02.709 INFO:teuthology.orchestra.run.smithi094.stdout: Birth: - 2023-11-24T06:07:02.709 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-24T06:07:02.772 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records in 2023-11-24T06:07:02.772 INFO:teuthology.orchestra.run.smithi094.stderr:1+0 records out 2023-11-24T06:07:02.772 INFO:teuthology.orchestra.run.smithi094.stderr:512 bytes copied, 9.2306e-05 s, 5.5 MB/s 2023-11-24T06:07:02.773 DEBUG:teuthology.orchestra.run.smithi094:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-24T06:07:02.832 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-24T06:07:02.832 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi091.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-24T06:07:02.832 INFO:tasks.ceph:Generating config... 2023-11-24T06:07:02.834 INFO:tasks.ceph:[client] client mount timeout = 600 2023-11-24T06:07:02.834 INFO:tasks.ceph:[client] debug client = 20 2023-11-24T06:07:02.834 INFO:tasks.ceph:[client] debug ms = 1 2023-11-24T06:07:02.834 INFO:tasks.ceph:[client] pid file = 2023-11-24T06:07:02.834 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-11-24T06:07:02.834 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-11-24T06:07:02.834 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-11-24T06:07:02.834 INFO:tasks.ceph:[mds] debug mds = 20 2023-11-24T06:07:02.834 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-11-24T06:07:02.834 INFO:tasks.ceph:[mds] debug ms = 1 2023-11-24T06:07:02.834 INFO:tasks.ceph:[mds] mds debug frag = True 2023-11-24T06:07:02.835 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-11-24T06:07:02.835 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-11-24T06:07:02.835 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-11-24T06:07:02.835 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-11-24T06:07:02.835 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-11-24T06:07:02.835 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-11-24T06:07:02.835 INFO:tasks.ceph:[mgr] debug client = 20 2023-11-24T06:07:02.835 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-11-24T06:07:02.835 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-24T06:07:02.835 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-24T06:07:02.835 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-24T06:07:02.835 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-24T06:07:02.835 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-24T06:07:02.836 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-11-24T06:07:02.836 INFO:tasks.ceph:[osd] bdev async discard = True 2023-11-24T06:07:02.836 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-11-24T06:07:02.836 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-11-24T06:07:02.837 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-24T06:07:02.837 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-24T06:07:02.837 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-11-24T06:07:02.837 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-11-24T06:07:02.837 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-24T06:07:02.837 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-24T06:07:02.837 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-11-24T06:07:02.837 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-24T06:07:02.837 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-24T06:07:02.838 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-24T06:07:02.838 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-24T06:07:02.838 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-24T06:07:02.838 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-11-24T06:07:02.838 INFO:tasks.ceph:Setting up mon.a... 2023-11-24T06:07:02.838 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-24T06:07:02.889 INFO:teuthology.orchestra.run.smithi091.stdout:creating /etc/ceph/ceph.keyring 2023-11-24T06:07:02.892 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-24T06:07:02.939 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-24T06:07:02.965 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.91'), ('mon.b', '172.21.15.94'), ('mon.c', '[v2:172.21.15.94:3301,v1:172.21.15.94:6790]')] 2023-11-24T06:07:02.965 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.91,172.21.15.94,[v2:172.21.15.94:3301,v1:172.21.15.94:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-11-24T06:07:02.966 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:02.966 DEBUG:teuthology.orchestra.run.smithi091:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-24T06:07:03.021 DEBUG:teuthology.orchestra.run.smithi091:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.91 --add b 172.21.15.94 --addv c '[v2:172.21.15.94:3301,v1:172.21.15.94:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:07:03.109 INFO:teuthology.orchestra.run.smithi091.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:07:03.109 INFO:teuthology.orchestra.run.smithi091.stdout:monmaptool: generated fsid f7c07073-6cd6-496f-812a-c57c86a2fbbb 2023-11-24T06:07:03.109 INFO:teuthology.orchestra.run.smithi091.stdout:setting min_mon_release = pacific 2023-11-24T06:07:03.109 INFO:teuthology.orchestra.run.smithi091.stdout:epoch 0 2023-11-24T06:07:03.109 INFO:teuthology.orchestra.run.smithi091.stdout:fsid f7c07073-6cd6-496f-812a-c57c86a2fbbb 2023-11-24T06:07:03.109 INFO:teuthology.orchestra.run.smithi091.stdout:last_changed 2023-11-24T06:07:03.107164+0000 2023-11-24T06:07:03.109 INFO:teuthology.orchestra.run.smithi091.stdout:created 2023-11-24T06:07:03.107164+0000 2023-11-24T06:07:03.109 INFO:teuthology.orchestra.run.smithi091.stdout:min_mon_release 16 (pacific) 2023-11-24T06:07:03.110 INFO:teuthology.orchestra.run.smithi091.stdout:election_strategy: 1 2023-11-24T06:07:03.110 INFO:teuthology.orchestra.run.smithi091.stdout:0: [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] mon.a 2023-11-24T06:07:03.110 INFO:teuthology.orchestra.run.smithi091.stdout:1: [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] mon.b 2023-11-24T06:07:03.110 INFO:teuthology.orchestra.run.smithi091.stdout:2: [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] mon.c 2023-11-24T06:07:03.110 INFO:teuthology.orchestra.run.smithi091.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-11-24T06:07:03.111 DEBUG:teuthology.orchestra.run.smithi091:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-24T06:07:03.127 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f7c07073-6cd6-496f-812a-c57c86a2fbbb... 2023-11-24T06:07:03.128 DEBUG:teuthology.orchestra.run.smithi091:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-24T06:07:03.170 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-24T06:07:03.212 INFO:teuthology.orchestra.run.smithi091.stdout:[global] 2023-11-24T06:07:03.213 INFO:teuthology.orchestra.run.smithi091.stdout: chdir = "" 2023-11-24T06:07:03.213 INFO:teuthology.orchestra.run.smithi091.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-24T06:07:03.213 INFO:teuthology.orchestra.run.smithi091.stdout: auth supported = cephx 2023-11-24T06:07:03.213 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.213 INFO:teuthology.orchestra.run.smithi091.stdout: filestore xattr use omap = true 2023-11-24T06:07:03.213 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.213 INFO:teuthology.orchestra.run.smithi091.stdout: mon clock drift allowed = 1.000 2023-11-24T06:07:03.213 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.214 INFO:teuthology.orchestra.run.smithi091.stdout: osd crush chooseleaf type = 0 2023-11-24T06:07:03.214 INFO:teuthology.orchestra.run.smithi091.stdout: auth debug = true 2023-11-24T06:07:03.214 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.214 INFO:teuthology.orchestra.run.smithi091.stdout: ms die on old message = true 2023-11-24T06:07:03.214 INFO:teuthology.orchestra.run.smithi091.stdout: ms die on bug = true 2023-11-24T06:07:03.214 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.214 INFO:teuthology.orchestra.run.smithi091.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-24T06:07:03.214 INFO:teuthology.orchestra.run.smithi091.stdout: mon pg warn max object skew = 0 2023-11-24T06:07:03.214 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.215 INFO:teuthology.orchestra.run.smithi091.stdout: # disable pg_autoscaler by default for new pools 2023-11-24T06:07:03.215 INFO:teuthology.orchestra.run.smithi091.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-24T06:07:03.215 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.215 INFO:teuthology.orchestra.run.smithi091.stdout: osd pool default size = 2 2023-11-24T06:07:03.215 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.215 INFO:teuthology.orchestra.run.smithi091.stdout: mon osd allow primary affinity = true 2023-11-24T06:07:03.215 INFO:teuthology.orchestra.run.smithi091.stdout: mon osd allow pg remap = true 2023-11-24T06:07:03.215 INFO:teuthology.orchestra.run.smithi091.stdout: mon warn on legacy crush tunables = false 2023-11-24T06:07:03.215 INFO:teuthology.orchestra.run.smithi091.stdout: mon warn on crush straw calc version zero = false 2023-11-24T06:07:03.215 INFO:teuthology.orchestra.run.smithi091.stdout: mon warn on no sortbitwise = false 2023-11-24T06:07:03.215 INFO:teuthology.orchestra.run.smithi091.stdout: mon warn on osd down out interval zero = false 2023-11-24T06:07:03.215 INFO:teuthology.orchestra.run.smithi091.stdout: mon warn on too few osds = false 2023-11-24T06:07:03.216 INFO:teuthology.orchestra.run.smithi091.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-24T06:07:03.216 INFO:teuthology.orchestra.run.smithi091.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-24T06:07:03.216 INFO:teuthology.orchestra.run.smithi091.stdout: mon_allow_pool_size_one = true 2023-11-24T06:07:03.216 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.216 INFO:teuthology.orchestra.run.smithi091.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-24T06:07:03.216 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.216 INFO:teuthology.orchestra.run.smithi091.stdout: osd default data pool replay window = 5 2023-11-24T06:07:03.216 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.216 INFO:teuthology.orchestra.run.smithi091.stdout: mon allow pool delete = true 2023-11-24T06:07:03.216 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.216 INFO:teuthology.orchestra.run.smithi091.stdout: mon cluster log file level = debug 2023-11-24T06:07:03.216 INFO:teuthology.orchestra.run.smithi091.stdout: debug asserts on shutdown = true 2023-11-24T06:07:03.217 INFO:teuthology.orchestra.run.smithi091.stdout: mon health detail to clog = false 2023-11-24T06:07:03.217 INFO:teuthology.orchestra.run.smithi091.stdout: mon host = "172.21.15.91,172.21.15.94,[v2:172.21.15.94:3301,v1:172.21.15.94:6790]" 2023-11-24T06:07:03.217 INFO:teuthology.orchestra.run.smithi091.stdout: ms die on skipped message = False 2023-11-24T06:07:03.217 INFO:teuthology.orchestra.run.smithi091.stdout: fsid = f7c07073-6cd6-496f-812a-c57c86a2fbbb 2023-11-24T06:07:03.217 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.217 INFO:teuthology.orchestra.run.smithi091.stdout:[osd] 2023-11-24T06:07:03.217 INFO:teuthology.orchestra.run.smithi091.stdout: osd journal size = 100 2023-11-24T06:07:03.217 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.217 INFO:teuthology.orchestra.run.smithi091.stdout: osd scrub load threshold = 5.0 2023-11-24T06:07:03.217 INFO:teuthology.orchestra.run.smithi091.stdout: osd scrub max interval = 600 2023-11-24T06:07:03.217 INFO:teuthology.orchestra.run.smithi091.stdout: osd mclock profile = high_recovery_ops 2023-11-24T06:07:03.217 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.218 INFO:teuthology.orchestra.run.smithi091.stdout: osd recover clone overlap = true 2023-11-24T06:07:03.218 INFO:teuthology.orchestra.run.smithi091.stdout: osd recovery max chunk = 1048576 2023-11-24T06:07:03.218 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.218 INFO:teuthology.orchestra.run.smithi091.stdout: osd debug shutdown = true 2023-11-24T06:07:03.218 INFO:teuthology.orchestra.run.smithi091.stdout: osd debug op order = true 2023-11-24T06:07:03.218 INFO:teuthology.orchestra.run.smithi091.stdout: osd debug verify stray on activate = true 2023-11-24T06:07:03.218 INFO:teuthology.orchestra.run.smithi091.stdout: osd debug trim objects = true 2023-11-24T06:07:03.218 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.218 INFO:teuthology.orchestra.run.smithi091.stdout: osd open classes on start = true 2023-11-24T06:07:03.218 INFO:teuthology.orchestra.run.smithi091.stdout: osd debug pg log writeout = true 2023-11-24T06:07:03.218 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.218 INFO:teuthology.orchestra.run.smithi091.stdout: osd deep scrub update digest min age = 30 2023-11-24T06:07:03.219 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.219 INFO:teuthology.orchestra.run.smithi091.stdout: osd map max advance = 10 2023-11-24T06:07:03.219 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.219 INFO:teuthology.orchestra.run.smithi091.stdout: journal zero on create = true 2023-11-24T06:07:03.219 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.219 INFO:teuthology.orchestra.run.smithi091.stdout: filestore ondisk finisher threads = 3 2023-11-24T06:07:03.219 INFO:teuthology.orchestra.run.smithi091.stdout: filestore apply finisher threads = 3 2023-11-24T06:07:03.219 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.219 INFO:teuthology.orchestra.run.smithi091.stdout: bdev debug aio = true 2023-11-24T06:07:03.219 INFO:teuthology.orchestra.run.smithi091.stdout: osd debug misdirected ops = true 2023-11-24T06:07:03.219 INFO:teuthology.orchestra.run.smithi091.stdout: bdev async discard = True 2023-11-24T06:07:03.219 INFO:teuthology.orchestra.run.smithi091.stdout: bdev enable discard = True 2023-11-24T06:07:03.220 INFO:teuthology.orchestra.run.smithi091.stdout: bluestore allocator = bitmap 2023-11-24T06:07:03.220 INFO:teuthology.orchestra.run.smithi091.stdout: bluestore block size = 96636764160 2023-11-24T06:07:03.220 INFO:teuthology.orchestra.run.smithi091.stdout: bluestore fsck on mount = True 2023-11-24T06:07:03.220 INFO:teuthology.orchestra.run.smithi091.stdout: debug bluefs = 1/20 2023-11-24T06:07:03.220 INFO:teuthology.orchestra.run.smithi091.stdout: debug bluestore = 1/20 2023-11-24T06:07:03.220 INFO:teuthology.orchestra.run.smithi091.stdout: debug ms = 1 2023-11-24T06:07:03.220 INFO:teuthology.orchestra.run.smithi091.stdout: debug osd = 20 2023-11-24T06:07:03.220 INFO:teuthology.orchestra.run.smithi091.stdout: debug rocksdb = 4/10 2023-11-24T06:07:03.220 INFO:teuthology.orchestra.run.smithi091.stdout: mon osd backfillfull_ratio = 0.85 2023-11-24T06:07:03.220 INFO:teuthology.orchestra.run.smithi091.stdout: mon osd full ratio = 0.9 2023-11-24T06:07:03.220 INFO:teuthology.orchestra.run.smithi091.stdout: mon osd nearfull ratio = 0.8 2023-11-24T06:07:03.220 INFO:teuthology.orchestra.run.smithi091.stdout: osd failsafe full ratio = 0.95 2023-11-24T06:07:03.221 INFO:teuthology.orchestra.run.smithi091.stdout: osd objectstore = bluestore 2023-11-24T06:07:03.221 INFO:teuthology.orchestra.run.smithi091.stdout: osd op complaint time = 180 2023-11-24T06:07:03.221 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.221 INFO:teuthology.orchestra.run.smithi091.stdout:[mgr] 2023-11-24T06:07:03.221 INFO:teuthology.orchestra.run.smithi091.stdout: debug ms = 1 2023-11-24T06:07:03.221 INFO:teuthology.orchestra.run.smithi091.stdout: debug mgr = 20 2023-11-24T06:07:03.221 INFO:teuthology.orchestra.run.smithi091.stdout: debug mon = 20 2023-11-24T06:07:03.221 INFO:teuthology.orchestra.run.smithi091.stdout: debug auth = 20 2023-11-24T06:07:03.221 INFO:teuthology.orchestra.run.smithi091.stdout: mon reweight min pgs per osd = 4 2023-11-24T06:07:03.221 INFO:teuthology.orchestra.run.smithi091.stdout: mon reweight min bytes per osd = 10 2023-11-24T06:07:03.222 INFO:teuthology.orchestra.run.smithi091.stdout: mgr/telemetry/nag = false 2023-11-24T06:07:03.222 INFO:teuthology.orchestra.run.smithi091.stdout: debug client = 20 2023-11-24T06:07:03.222 INFO:teuthology.orchestra.run.smithi091.stdout: debug finisher = 20 2023-11-24T06:07:03.222 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.222 INFO:teuthology.orchestra.run.smithi091.stdout:[mon] 2023-11-24T06:07:03.222 INFO:teuthology.orchestra.run.smithi091.stdout: debug ms = 1 2023-11-24T06:07:03.222 INFO:teuthology.orchestra.run.smithi091.stdout: debug mon = 20 2023-11-24T06:07:03.222 INFO:teuthology.orchestra.run.smithi091.stdout: debug paxos = 20 2023-11-24T06:07:03.222 INFO:teuthology.orchestra.run.smithi091.stdout: debug auth = 20 2023-11-24T06:07:03.222 INFO:teuthology.orchestra.run.smithi091.stdout: mon data avail warn = 5 2023-11-24T06:07:03.222 INFO:teuthology.orchestra.run.smithi091.stdout: mon mgr mkfs grace = 240 2023-11-24T06:07:03.222 INFO:teuthology.orchestra.run.smithi091.stdout: mon reweight min pgs per osd = 4 2023-11-24T06:07:03.223 INFO:teuthology.orchestra.run.smithi091.stdout: mon osd reporter subtree level = osd 2023-11-24T06:07:03.223 INFO:teuthology.orchestra.run.smithi091.stdout: mon osd prime pg temp = true 2023-11-24T06:07:03.223 INFO:teuthology.orchestra.run.smithi091.stdout: mon reweight min bytes per osd = 10 2023-11-24T06:07:03.223 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.223 INFO:teuthology.orchestra.run.smithi091.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-24T06:07:03.223 INFO:teuthology.orchestra.run.smithi091.stdout: auth mon ticket ttl = 660 # 11m 2023-11-24T06:07:03.223 INFO:teuthology.orchestra.run.smithi091.stdout: auth service ticket ttl = 240 # 4m 2023-11-24T06:07:03.223 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.223 INFO:teuthology.orchestra.run.smithi091.stdout: # don't complain about insecure global_id in the test suite 2023-11-24T06:07:03.223 INFO:teuthology.orchestra.run.smithi091.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-24T06:07:03.223 INFO:teuthology.orchestra.run.smithi091.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-24T06:07:03.223 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.223 INFO:teuthology.orchestra.run.smithi091.stdout: # 1m isn't quite enough 2023-11-24T06:07:03.224 INFO:teuthology.orchestra.run.smithi091.stdout: mon_down_mkfs_grace = 2m 2023-11-24T06:07:03.224 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.224 INFO:teuthology.orchestra.run.smithi091.stdout: mon_warn_on_filestore_osds = false 2023-11-24T06:07:03.224 INFO:teuthology.orchestra.run.smithi091.stdout: mon op complaint time = 120 2023-11-24T06:07:03.224 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:03.224 INFO:teuthology.orchestra.run.smithi091.stdout:[client] 2023-11-24T06:07:03.224 INFO:teuthology.orchestra.run.smithi091.stdout: rgw cache enabled = true 2023-11-24T06:07:03.224 INFO:teuthology.orchestra.run.smithi091.stdout: rgw enable ops log = true 2023-11-24T06:07:03.224 INFO:teuthology.orchestra.run.smithi091.stdout: rgw enable usage log = true 2023-11-24T06:07:03.224 INFO:teuthology.orchestra.run.smithi091.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-24T06:07:03.224 INFO:teuthology.orchestra.run.smithi091.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-24T06:07:03.224 INFO:teuthology.orchestra.run.smithi091.stdout: client mount timeout = 600 2023-11-24T06:07:03.225 INFO:teuthology.orchestra.run.smithi091.stdout: debug client = 20 2023-11-24T06:07:03.225 INFO:teuthology.orchestra.run.smithi091.stdout: debug ms = 1 2023-11-24T06:07:03.225 INFO:teuthology.orchestra.run.smithi091.stdout: pid file = "" 2023-11-24T06:07:03.225 INFO:teuthology.orchestra.run.smithi091.stdout: rados mon op timeout = 900 2023-11-24T06:07:03.225 INFO:teuthology.orchestra.run.smithi091.stdout: rados osd op timeout = 900 2023-11-24T06:07:03.225 INFO:teuthology.orchestra.run.smithi091.stdout:[mon.a] 2023-11-24T06:07:03.225 INFO:teuthology.orchestra.run.smithi091.stdout:[mon.b] 2023-11-24T06:07:03.225 INFO:teuthology.orchestra.run.smithi091.stdout:[mon.c] 2023-11-24T06:07:03.225 INFO:teuthology.orchestra.run.smithi091.stdout:[mds.a] 2023-11-24T06:07:03.225 INFO:teuthology.orchestra.run.smithi091.stdout:[mds.c] 2023-11-24T06:07:03.225 INFO:teuthology.orchestra.run.smithi091.stdout:[mds.b] 2023-11-24T06:07:03.225 INFO:teuthology.orchestra.run.smithi091.stdout:[mds.d] 2023-11-24T06:07:03.225 INFO:teuthology.orchestra.run.smithi091.stdout:[mds] 2023-11-24T06:07:03.226 INFO:teuthology.orchestra.run.smithi091.stdout: debug mds = 20 2023-11-24T06:07:03.226 INFO:teuthology.orchestra.run.smithi091.stdout: debug mds balancer = 20 2023-11-24T06:07:03.226 INFO:teuthology.orchestra.run.smithi091.stdout: debug ms = 1 2023-11-24T06:07:03.226 INFO:teuthology.orchestra.run.smithi091.stdout: mds debug frag = True 2023-11-24T06:07:03.226 INFO:teuthology.orchestra.run.smithi091.stdout: mds debug scatterstat = True 2023-11-24T06:07:03.226 INFO:teuthology.orchestra.run.smithi091.stdout: mds op complaint time = 180 2023-11-24T06:07:03.226 INFO:teuthology.orchestra.run.smithi091.stdout: mds verify scatter = True 2023-11-24T06:07:03.226 INFO:teuthology.orchestra.run.smithi091.stdout: osd op complaint time = 180 2023-11-24T06:07:03.226 INFO:teuthology.orchestra.run.smithi091.stdout: rados mon op timeout = 900 2023-11-24T06:07:03.226 INFO:teuthology.orchestra.run.smithi091.stdout: rados osd op timeout = 900 2023-11-24T06:07:03.227 INFO:teuthology.orchestra.run.smithi094.stdout:[global] 2023-11-24T06:07:03.227 INFO:teuthology.orchestra.run.smithi094.stdout: chdir = "" 2023-11-24T06:07:03.227 INFO:teuthology.orchestra.run.smithi094.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-24T06:07:03.227 INFO:teuthology.orchestra.run.smithi094.stdout: auth supported = cephx 2023-11-24T06:07:03.227 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.227 INFO:teuthology.orchestra.run.smithi094.stdout: filestore xattr use omap = true 2023-11-24T06:07:03.227 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.227 INFO:teuthology.orchestra.run.smithi094.stdout: mon clock drift allowed = 1.000 2023-11-24T06:07:03.228 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.228 INFO:teuthology.orchestra.run.smithi094.stdout: osd crush chooseleaf type = 0 2023-11-24T06:07:03.228 INFO:teuthology.orchestra.run.smithi094.stdout: auth debug = true 2023-11-24T06:07:03.228 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.228 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on old message = true 2023-11-24T06:07:03.228 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on bug = true 2023-11-24T06:07:03.228 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.228 INFO:teuthology.orchestra.run.smithi094.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-24T06:07:03.228 INFO:teuthology.orchestra.run.smithi094.stdout: mon pg warn max object skew = 0 2023-11-24T06:07:03.228 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.228 INFO:teuthology.orchestra.run.smithi094.stdout: # disable pg_autoscaler by default for new pools 2023-11-24T06:07:03.228 INFO:teuthology.orchestra.run.smithi094.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-24T06:07:03.229 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.229 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default size = 2 2023-11-24T06:07:03.229 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.229 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow primary affinity = true 2023-11-24T06:07:03.229 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd allow pg remap = true 2023-11-24T06:07:03.229 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on legacy crush tunables = false 2023-11-24T06:07:03.229 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on crush straw calc version zero = false 2023-11-24T06:07:03.229 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on no sortbitwise = false 2023-11-24T06:07:03.229 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on osd down out interval zero = false 2023-11-24T06:07:03.229 INFO:teuthology.orchestra.run.smithi094.stdout: mon warn on too few osds = false 2023-11-24T06:07:03.229 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-24T06:07:03.229 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-24T06:07:03.230 INFO:teuthology.orchestra.run.smithi094.stdout: mon_allow_pool_size_one = true 2023-11-24T06:07:03.230 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.230 INFO:teuthology.orchestra.run.smithi094.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-24T06:07:03.230 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.230 INFO:teuthology.orchestra.run.smithi094.stdout: osd default data pool replay window = 5 2023-11-24T06:07:03.230 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.230 INFO:teuthology.orchestra.run.smithi094.stdout: mon allow pool delete = true 2023-11-24T06:07:03.230 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.230 INFO:teuthology.orchestra.run.smithi094.stdout: mon cluster log file level = debug 2023-11-24T06:07:03.230 INFO:teuthology.orchestra.run.smithi094.stdout: debug asserts on shutdown = true 2023-11-24T06:07:03.231 INFO:teuthology.orchestra.run.smithi094.stdout: mon health detail to clog = false 2023-11-24T06:07:03.231 INFO:teuthology.orchestra.run.smithi094.stdout: mon host = "172.21.15.91,172.21.15.94,[v2:172.21.15.94:3301,v1:172.21.15.94:6790]" 2023-11-24T06:07:03.231 INFO:teuthology.orchestra.run.smithi094.stdout: ms die on skipped message = False 2023-11-24T06:07:03.231 INFO:teuthology.orchestra.run.smithi094.stdout: fsid = f7c07073-6cd6-496f-812a-c57c86a2fbbb 2023-11-24T06:07:03.231 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.231 INFO:teuthology.orchestra.run.smithi094.stdout:[osd] 2023-11-24T06:07:03.231 INFO:teuthology.orchestra.run.smithi094.stdout: osd journal size = 100 2023-11-24T06:07:03.231 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.231 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub load threshold = 5.0 2023-11-24T06:07:03.231 INFO:teuthology.orchestra.run.smithi094.stdout: osd scrub max interval = 600 2023-11-24T06:07:03.231 INFO:teuthology.orchestra.run.smithi094.stdout: osd mclock profile = high_recovery_ops 2023-11-24T06:07:03.232 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.232 INFO:teuthology.orchestra.run.smithi094.stdout: osd recover clone overlap = true 2023-11-24T06:07:03.232 INFO:teuthology.orchestra.run.smithi094.stdout: osd recovery max chunk = 1048576 2023-11-24T06:07:03.232 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.232 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug shutdown = true 2023-11-24T06:07:03.232 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug op order = true 2023-11-24T06:07:03.232 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug verify stray on activate = true 2023-11-24T06:07:03.232 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug trim objects = true 2023-11-24T06:07:03.232 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.232 INFO:teuthology.orchestra.run.smithi094.stdout: osd open classes on start = true 2023-11-24T06:07:03.232 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug pg log writeout = true 2023-11-24T06:07:03.232 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.233 INFO:teuthology.orchestra.run.smithi094.stdout: osd deep scrub update digest min age = 30 2023-11-24T06:07:03.233 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.233 INFO:teuthology.orchestra.run.smithi094.stdout: osd map max advance = 10 2023-11-24T06:07:03.233 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.233 INFO:teuthology.orchestra.run.smithi094.stdout: journal zero on create = true 2023-11-24T06:07:03.233 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.233 INFO:teuthology.orchestra.run.smithi094.stdout: filestore ondisk finisher threads = 3 2023-11-24T06:07:03.233 INFO:teuthology.orchestra.run.smithi094.stdout: filestore apply finisher threads = 3 2023-11-24T06:07:03.233 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.233 INFO:teuthology.orchestra.run.smithi094.stdout: bdev debug aio = true 2023-11-24T06:07:03.233 INFO:teuthology.orchestra.run.smithi094.stdout: osd debug misdirected ops = true 2023-11-24T06:07:03.233 INFO:teuthology.orchestra.run.smithi094.stdout: bdev async discard = True 2023-11-24T06:07:03.234 INFO:teuthology.orchestra.run.smithi094.stdout: bdev enable discard = True 2023-11-24T06:07:03.234 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore allocator = bitmap 2023-11-24T06:07:03.234 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore block size = 96636764160 2023-11-24T06:07:03.234 INFO:teuthology.orchestra.run.smithi094.stdout: bluestore fsck on mount = True 2023-11-24T06:07:03.234 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluefs = 1/20 2023-11-24T06:07:03.234 INFO:teuthology.orchestra.run.smithi094.stdout: debug bluestore = 1/20 2023-11-24T06:07:03.234 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2023-11-24T06:07:03.234 INFO:teuthology.orchestra.run.smithi094.stdout: debug osd = 20 2023-11-24T06:07:03.234 INFO:teuthology.orchestra.run.smithi094.stdout: debug rocksdb = 4/10 2023-11-24T06:07:03.234 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd backfillfull_ratio = 0.85 2023-11-24T06:07:03.234 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd full ratio = 0.9 2023-11-24T06:07:03.234 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd nearfull ratio = 0.8 2023-11-24T06:07:03.235 INFO:teuthology.orchestra.run.smithi094.stdout: osd failsafe full ratio = 0.95 2023-11-24T06:07:03.235 INFO:teuthology.orchestra.run.smithi094.stdout: osd objectstore = bluestore 2023-11-24T06:07:03.235 INFO:teuthology.orchestra.run.smithi094.stdout: osd op complaint time = 180 2023-11-24T06:07:03.235 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.235 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr] 2023-11-24T06:07:03.235 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2023-11-24T06:07:03.235 INFO:teuthology.orchestra.run.smithi094.stdout: debug mgr = 20 2023-11-24T06:07:03.235 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2023-11-24T06:07:03.235 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2023-11-24T06:07:03.235 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2023-11-24T06:07:03.235 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2023-11-24T06:07:03.235 INFO:teuthology.orchestra.run.smithi094.stdout: mgr/telemetry/nag = false 2023-11-24T06:07:03.236 INFO:teuthology.orchestra.run.smithi094.stdout: debug client = 20 2023-11-24T06:07:03.236 INFO:teuthology.orchestra.run.smithi094.stdout: debug finisher = 20 2023-11-24T06:07:03.236 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.236 INFO:teuthology.orchestra.run.smithi094.stdout:[mon] 2023-11-24T06:07:03.236 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2023-11-24T06:07:03.236 INFO:teuthology.orchestra.run.smithi094.stdout: debug mon = 20 2023-11-24T06:07:03.236 INFO:teuthology.orchestra.run.smithi094.stdout: debug paxos = 20 2023-11-24T06:07:03.236 INFO:teuthology.orchestra.run.smithi094.stdout: debug auth = 20 2023-11-24T06:07:03.236 INFO:teuthology.orchestra.run.smithi094.stdout: mon data avail warn = 5 2023-11-24T06:07:03.236 INFO:teuthology.orchestra.run.smithi094.stdout: mon mgr mkfs grace = 240 2023-11-24T06:07:03.236 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min pgs per osd = 4 2023-11-24T06:07:03.236 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd reporter subtree level = osd 2023-11-24T06:07:03.236 INFO:teuthology.orchestra.run.smithi094.stdout: mon osd prime pg temp = true 2023-11-24T06:07:03.237 INFO:teuthology.orchestra.run.smithi094.stdout: mon reweight min bytes per osd = 10 2023-11-24T06:07:03.237 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.237 INFO:teuthology.orchestra.run.smithi094.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-24T06:07:03.237 INFO:teuthology.orchestra.run.smithi094.stdout: auth mon ticket ttl = 660 # 11m 2023-11-24T06:07:03.237 INFO:teuthology.orchestra.run.smithi094.stdout: auth service ticket ttl = 240 # 4m 2023-11-24T06:07:03.237 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.237 INFO:teuthology.orchestra.run.smithi094.stdout: # don't complain about insecure global_id in the test suite 2023-11-24T06:07:03.237 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-24T06:07:03.237 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-24T06:07:03.237 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.237 INFO:teuthology.orchestra.run.smithi094.stdout: # 1m isn't quite enough 2023-11-24T06:07:03.237 INFO:teuthology.orchestra.run.smithi094.stdout: mon_down_mkfs_grace = 2m 2023-11-24T06:07:03.237 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.238 INFO:teuthology.orchestra.run.smithi094.stdout: mon_warn_on_filestore_osds = false 2023-11-24T06:07:03.238 INFO:teuthology.orchestra.run.smithi094.stdout: mon op complaint time = 120 2023-11-24T06:07:03.238 INFO:teuthology.orchestra.run.smithi094.stdout: 2023-11-24T06:07:03.238 INFO:teuthology.orchestra.run.smithi094.stdout:[client] 2023-11-24T06:07:03.238 INFO:teuthology.orchestra.run.smithi094.stdout: rgw cache enabled = true 2023-11-24T06:07:03.238 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable ops log = true 2023-11-24T06:07:03.238 INFO:teuthology.orchestra.run.smithi094.stdout: rgw enable usage log = true 2023-11-24T06:07:03.238 INFO:teuthology.orchestra.run.smithi094.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-24T06:07:03.238 INFO:teuthology.orchestra.run.smithi094.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-24T06:07:03.238 INFO:teuthology.orchestra.run.smithi094.stdout: client mount timeout = 600 2023-11-24T06:07:03.238 INFO:teuthology.orchestra.run.smithi094.stdout: debug client = 20 2023-11-24T06:07:03.238 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2023-11-24T06:07:03.239 INFO:teuthology.orchestra.run.smithi094.stdout: pid file = "" 2023-11-24T06:07:03.239 INFO:teuthology.orchestra.run.smithi094.stdout: rados mon op timeout = 900 2023-11-24T06:07:03.239 INFO:teuthology.orchestra.run.smithi094.stdout: rados osd op timeout = 900 2023-11-24T06:07:03.239 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.a] 2023-11-24T06:07:03.239 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.b] 2023-11-24T06:07:03.239 INFO:teuthology.orchestra.run.smithi094.stdout:[mon.c] 2023-11-24T06:07:03.239 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.a] 2023-11-24T06:07:03.239 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.c] 2023-11-24T06:07:03.239 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.b] 2023-11-24T06:07:03.239 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.d] 2023-11-24T06:07:03.239 INFO:teuthology.orchestra.run.smithi094.stdout:[mds] 2023-11-24T06:07:03.239 INFO:teuthology.orchestra.run.smithi094.stdout: debug mds = 20 2023-11-24T06:07:03.240 INFO:teuthology.orchestra.run.smithi094.stdout: debug mds balancer = 20 2023-11-24T06:07:03.240 INFO:teuthology.orchestra.run.smithi094.stdout: debug ms = 1 2023-11-24T06:07:03.240 INFO:teuthology.orchestra.run.smithi094.stdout: mds debug frag = True 2023-11-24T06:07:03.240 INFO:teuthology.orchestra.run.smithi094.stdout: mds debug scatterstat = True 2023-11-24T06:07:03.240 INFO:teuthology.orchestra.run.smithi094.stdout: mds op complaint time = 180 2023-11-24T06:07:03.240 INFO:teuthology.orchestra.run.smithi094.stdout: mds verify scatter = True 2023-11-24T06:07:03.240 INFO:teuthology.orchestra.run.smithi094.stdout: osd op complaint time = 180 2023-11-24T06:07:03.240 INFO:teuthology.orchestra.run.smithi094.stdout: rados mon op timeout = 900 2023-11-24T06:07:03.240 INFO:teuthology.orchestra.run.smithi094.stdout: rados osd op timeout = 900 2023-11-24T06:07:03.241 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-24T06:07:03.242 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-24T06:07:03.339 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-24T06:07:03.339 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:03.339 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-24T06:07:03.355 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:03.356 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-24T06:07:03.412 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi091.front.sepia.ceph.com 2023-11-24T06:07:03.412 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:03.412 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-24T06:07:03.412 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-24T06:07:03.488 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:03.488 DEBUG:teuthology.orchestra.run.smithi091:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:07:03.545 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi094.front.sepia.ceph.com 2023-11-24T06:07:03.545 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:03.545 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-24T06:07:03.545 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-24T06:07:03.581 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:03.581 DEBUG:teuthology.orchestra.run.smithi094:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:07:03.637 INFO:tasks.ceph:Setting up mon nodes... 2023-11-24T06:07:03.638 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-24T06:07:03.638 DEBUG:teuthology.orchestra.run.smithi091:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-11-24T06:07:03.691 INFO:teuthology.orchestra.run.smithi091.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-11-24T06:07:03.694 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-24T06:07:03.768 INFO:teuthology.orchestra.run.smithi094.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-24T06:07:03.771 INFO:tasks.ceph:Setting up mds nodes... 2023-11-24T06:07:03.771 DEBUG:teuthology.orchestra.run.smithi091:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-11-24T06:07:03.825 INFO:teuthology.orchestra.run.smithi091.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-11-24T06:07:03.827 DEBUG:teuthology.orchestra.run.smithi091:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-11-24T06:07:03.852 DEBUG:teuthology.orchestra.run.smithi091:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-11-24T06:07:03.945 INFO:teuthology.orchestra.run.smithi091.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-11-24T06:07:03.947 DEBUG:teuthology.orchestra.run.smithi091:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-11-24T06:07:03.973 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-11-24T06:07:04.031 INFO:teuthology.orchestra.run.smithi094.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-11-24T06:07:04.034 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-11-24T06:07:04.059 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-11-24T06:07:04.154 INFO:teuthology.orchestra.run.smithi094.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-11-24T06:07:04.157 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-11-24T06:07:04.182 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-24T06:07:04.182 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-24T06:07:04.225 INFO:teuthology.orchestra.run.smithi091.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-24T06:07:04.238 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-11-24T06:07:04.320 INFO:teuthology.orchestra.run.smithi091.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-11-24T06:07:04.333 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-11-24T06:07:04.376 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-11-24T06:07:04.389 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-11-24T06:07:04.472 INFO:teuthology.orchestra.run.smithi094.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-11-24T06:07:04.484 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-24T06:07:04.484 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi091.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi094.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-11-24T06:07:04.484 DEBUG:teuthology.orchestra.run.smithi091:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-24T06:07:04.511 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-24T06:07:04.511 INFO:tasks.ceph:role: osd.0 2023-11-24T06:07:04.511 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi091.front.sepia.ceph.com 2023-11-24T06:07:04.511 DEBUG:teuthology.orchestra.run.smithi091:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-24T06:07:04.584 INFO:teuthology.orchestra.run.smithi091.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T06:07:04.584 INFO:teuthology.orchestra.run.smithi091.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T06:07:04.584 INFO:teuthology.orchestra.run.smithi091.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T06:07:04.584 INFO:teuthology.orchestra.run.smithi091.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T06:07:04.584 INFO:teuthology.orchestra.run.smithi091.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T06:07:04.584 INFO:teuthology.orchestra.run.smithi091.stdout: = sunit=0 swidth=0 blks 2023-11-24T06:07:04.584 INFO:teuthology.orchestra.run.smithi091.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T06:07:04.584 INFO:teuthology.orchestra.run.smithi091.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T06:07:04.584 INFO:teuthology.orchestra.run.smithi091.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T06:07:04.584 INFO:teuthology.orchestra.run.smithi091.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T06:07:05.015 INFO:teuthology.orchestra.run.smithi091.stdout:Discarding blocks...Done. 2023-11-24T06:07:05.017 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi091.front.sepia.ceph.com -o noatime 2023-11-24T06:07:05.017 DEBUG:teuthology.orchestra.run.smithi091:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-24T06:07:05.204 DEBUG:teuthology.orchestra.run.smithi091:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-24T06:07:05.237 DEBUG:teuthology.orchestra.run.smithi091:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-24T06:07:05.304 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-24T06:07:05.304 INFO:tasks.ceph:role: osd.1 2023-11-24T06:07:05.305 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi091.front.sepia.ceph.com 2023-11-24T06:07:05.305 DEBUG:teuthology.orchestra.run.smithi091:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-24T06:07:05.378 INFO:teuthology.orchestra.run.smithi091.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T06:07:05.378 INFO:teuthology.orchestra.run.smithi091.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T06:07:05.378 INFO:teuthology.orchestra.run.smithi091.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T06:07:05.378 INFO:teuthology.orchestra.run.smithi091.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T06:07:05.378 INFO:teuthology.orchestra.run.smithi091.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T06:07:05.378 INFO:teuthology.orchestra.run.smithi091.stdout: = sunit=0 swidth=0 blks 2023-11-24T06:07:05.378 INFO:teuthology.orchestra.run.smithi091.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T06:07:05.379 INFO:teuthology.orchestra.run.smithi091.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T06:07:05.379 INFO:teuthology.orchestra.run.smithi091.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T06:07:05.379 INFO:teuthology.orchestra.run.smithi091.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T06:07:05.803 INFO:teuthology.orchestra.run.smithi091.stdout:Discarding blocks...Done. 2023-11-24T06:07:05.805 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi091.front.sepia.ceph.com -o noatime 2023-11-24T06:07:05.805 DEBUG:teuthology.orchestra.run.smithi091:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-24T06:07:05.914 DEBUG:teuthology.orchestra.run.smithi091:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-24T06:07:05.944 DEBUG:teuthology.orchestra.run.smithi091:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-24T06:07:06.011 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-24T06:07:06.011 INFO:tasks.ceph:role: osd.2 2023-11-24T06:07:06.011 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi091.front.sepia.ceph.com 2023-11-24T06:07:06.011 DEBUG:teuthology.orchestra.run.smithi091:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-24T06:07:06.085 INFO:teuthology.orchestra.run.smithi091.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T06:07:06.085 INFO:teuthology.orchestra.run.smithi091.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T06:07:06.085 INFO:teuthology.orchestra.run.smithi091.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T06:07:06.085 INFO:teuthology.orchestra.run.smithi091.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T06:07:06.085 INFO:teuthology.orchestra.run.smithi091.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T06:07:06.085 INFO:teuthology.orchestra.run.smithi091.stdout: = sunit=0 swidth=0 blks 2023-11-24T06:07:06.085 INFO:teuthology.orchestra.run.smithi091.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T06:07:06.085 INFO:teuthology.orchestra.run.smithi091.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T06:07:06.085 INFO:teuthology.orchestra.run.smithi091.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T06:07:06.086 INFO:teuthology.orchestra.run.smithi091.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T06:07:06.533 INFO:teuthology.orchestra.run.smithi091.stdout:Discarding blocks...Done. 2023-11-24T06:07:06.534 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi091.front.sepia.ceph.com -o noatime 2023-11-24T06:07:06.535 DEBUG:teuthology.orchestra.run.smithi091:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-24T06:07:06.701 DEBUG:teuthology.orchestra.run.smithi091:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-24T06:07:06.731 DEBUG:teuthology.orchestra.run.smithi091:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-11-24T06:07:06.797 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-11-24T06:07:06.797 INFO:tasks.ceph:role: osd.3 2023-11-24T06:07:06.797 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi091.front.sepia.ceph.com 2023-11-24T06:07:06.797 DEBUG:teuthology.orchestra.run.smithi091:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-24T06:07:06.871 INFO:teuthology.orchestra.run.smithi091.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T06:07:06.871 INFO:teuthology.orchestra.run.smithi091.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T06:07:06.871 INFO:teuthology.orchestra.run.smithi091.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T06:07:06.871 INFO:teuthology.orchestra.run.smithi091.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T06:07:06.872 INFO:teuthology.orchestra.run.smithi091.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T06:07:06.872 INFO:teuthology.orchestra.run.smithi091.stdout: = sunit=0 swidth=0 blks 2023-11-24T06:07:06.872 INFO:teuthology.orchestra.run.smithi091.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T06:07:06.872 INFO:teuthology.orchestra.run.smithi091.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T06:07:06.872 INFO:teuthology.orchestra.run.smithi091.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T06:07:06.872 INFO:teuthology.orchestra.run.smithi091.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T06:07:07.320 INFO:teuthology.orchestra.run.smithi091.stdout:Discarding blocks...Done. 2023-11-24T06:07:07.321 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi091.front.sepia.ceph.com -o noatime 2023-11-24T06:07:07.322 DEBUG:teuthology.orchestra.run.smithi091:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-11-24T06:07:07.506 DEBUG:teuthology.orchestra.run.smithi091:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-11-24T06:07:07.538 DEBUG:teuthology.orchestra.run.smithi091:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:07:07.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:07.645+0000 7fbbf26a8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-11-24T06:07:07.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:07.646+0000 7fbbf26a8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-11-24T06:07:07.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:07.646+0000 7fbbf26a8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-24T06:07:07.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:07.646+0000 7fbbf26a8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-24T06:07:07.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:07.646+0000 7fbbf26a8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-11-24T06:07:07.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:07.646+0000 7fbbf26a8640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-11-24T06:07:10.511 DEBUG:teuthology.orchestra.run.smithi091:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-24T06:07:10.543 DEBUG:teuthology.orchestra.run.smithi091:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:07:10.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:10.630+0000 7fcfaa177640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-11-24T06:07:10.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:10.631+0000 7fcfaa177640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-11-24T06:07:10.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:10.631+0000 7fcfaa177640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-24T06:07:10.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:10.631+0000 7fcfaa177640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-24T06:07:10.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:10.631+0000 7fcfaa177640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-11-24T06:07:10.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:10.631+0000 7fcfaa177640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-11-24T06:07:13.498 DEBUG:teuthology.orchestra.run.smithi091:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-24T06:07:13.529 DEBUG:teuthology.orchestra.run.smithi091:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:07:13.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:13.619+0000 7ff711fb5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-11-24T06:07:13.623 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:13.620+0000 7ff711fb5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-11-24T06:07:13.623 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:13.620+0000 7ff711fb5640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-24T06:07:13.623 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:13.620+0000 7ff711fb5640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-24T06:07:13.623 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:13.620+0000 7ff711fb5640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-11-24T06:07:13.623 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:13.620+0000 7ff711fb5640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-11-24T06:07:16.488 DEBUG:teuthology.orchestra.run.smithi091:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-24T06:07:16.519 DEBUG:teuthology.orchestra.run.smithi091:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:07:16.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:16.608+0000 7fc9f5b34640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-11-24T06:07:16.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:16.608+0000 7fc9f5b34640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-11-24T06:07:16.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:16.608+0000 7fc9f5b34640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-24T06:07:16.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:16.608+0000 7fc9f5b34640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-24T06:07:16.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:16.608+0000 7fc9f5b34640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-11-24T06:07:16.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:16.608+0000 7fc9f5b34640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-11-24T06:07:19.471 DEBUG:teuthology.orchestra.run.smithi091:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-11-24T06:07:19.501 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-11-24T06:07:19.570 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-24T06:07:19.570 INFO:tasks.ceph:role: osd.4 2023-11-24T06:07:19.570 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com 2023-11-24T06:07:19.570 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-24T06:07:19.599 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T06:07:19.599 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T06:07:19.599 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T06:07:19.599 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T06:07:19.599 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T06:07:19.599 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2023-11-24T06:07:19.599 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T06:07:19.600 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T06:07:19.600 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T06:07:19.600 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T06:07:19.623 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2023-11-24T06:07:19.670 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2023-11-24T06:07:19.670 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-11-24T06:07:19.755 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-11-24T06:07:19.827 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-11-24T06:07:19.894 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-24T06:07:19.895 INFO:tasks.ceph:role: osd.5 2023-11-24T06:07:19.895 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com 2023-11-24T06:07:19.895 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-24T06:07:19.960 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T06:07:19.960 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T06:07:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T06:07:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T06:07:19.961 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T06:07:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2023-11-24T06:07:19.961 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T06:07:19.961 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T06:07:19.961 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T06:07:19.961 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T06:07:19.986 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2023-11-24T06:07:19.987 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2023-11-24T06:07:19.987 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-11-24T06:07:20.066 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-11-24T06:07:20.137 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-11-24T06:07:20.202 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-24T06:07:20.203 INFO:tasks.ceph:role: osd.6 2023-11-24T06:07:20.203 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com 2023-11-24T06:07:20.203 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-24T06:07:20.269 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T06:07:20.269 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T06:07:20.269 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T06:07:20.269 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T06:07:20.269 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T06:07:20.269 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2023-11-24T06:07:20.270 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T06:07:20.270 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T06:07:20.270 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T06:07:20.270 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T06:07:20.294 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2023-11-24T06:07:20.296 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2023-11-24T06:07:20.296 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-11-24T06:07:20.375 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-11-24T06:07:20.447 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-11-24T06:07:20.514 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-11-24T06:07:20.514 INFO:tasks.ceph:role: osd.7 2023-11-24T06:07:20.515 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi094.front.sepia.ceph.com 2023-11-24T06:07:20.515 DEBUG:teuthology.orchestra.run.smithi094:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-11-24T06:07:20.581 INFO:teuthology.orchestra.run.smithi094.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-11-24T06:07:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-24T06:07:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-24T06:07:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-24T06:07:20.581 INFO:teuthology.orchestra.run.smithi094.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-24T06:07:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: = sunit=0 swidth=0 blks 2023-11-24T06:07:20.581 INFO:teuthology.orchestra.run.smithi094.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-24T06:07:20.581 INFO:teuthology.orchestra.run.smithi094.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-24T06:07:20.581 INFO:teuthology.orchestra.run.smithi094.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-24T06:07:20.581 INFO:teuthology.orchestra.run.smithi094.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-24T06:07:20.605 INFO:teuthology.orchestra.run.smithi094.stdout:Discarding blocks...Done. 2023-11-24T06:07:20.607 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi094.front.sepia.ceph.com -o noatime 2023-11-24T06:07:20.607 DEBUG:teuthology.orchestra.run.smithi094:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-11-24T06:07:20.686 DEBUG:teuthology.orchestra.run.smithi094:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-11-24T06:07:20.757 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:07:20.864 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:20.860+0000 7f4b8df77640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-11-24T06:07:20.864 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:20.861+0000 7f4b8df77640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-11-24T06:07:20.864 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:20.861+0000 7f4b8df77640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-24T06:07:20.865 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:20.861+0000 7f4b8df77640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-24T06:07:20.865 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:20.861+0000 7f4b8df77640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-11-24T06:07:20.865 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:20.861+0000 7f4b8df77640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-11-24T06:07:23.733 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-11-24T06:07:23.765 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:07:23.856 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:23.853+0000 7fd3d44b6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-11-24T06:07:23.856 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:23.853+0000 7fd3d44b6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-11-24T06:07:23.856 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:23.853+0000 7fd3d44b6640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-24T06:07:23.857 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:23.853+0000 7fd3d44b6640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-24T06:07:23.857 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:23.853+0000 7fd3d44b6640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-11-24T06:07:23.857 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:23.853+0000 7fd3d44b6640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-11-24T06:07:26.723 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-11-24T06:07:26.755 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:07:26.847 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:26.844+0000 7fb2cc845640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-11-24T06:07:26.847 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:26.844+0000 7fb2cc845640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-11-24T06:07:26.848 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:26.844+0000 7fb2cc845640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-24T06:07:26.848 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:26.844+0000 7fb2cc845640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-24T06:07:26.848 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:26.844+0000 7fb2cc845640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-11-24T06:07:26.848 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:26.844+0000 7fb2cc845640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-11-24T06:07:29.719 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-11-24T06:07:29.750 DEBUG:teuthology.orchestra.run.smithi094:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:07:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:29.837+0000 7fe4dd5dc640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-11-24T06:07:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:29.837+0000 7fe4dd5dc640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-11-24T06:07:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:29.837+0000 7fe4dd5dc640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-24T06:07:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:29.837+0000 7fe4dd5dc640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-24T06:07:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:29.837+0000 7fe4dd5dc640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-11-24T06:07:29.841 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:29.838+0000 7fe4dd5dc640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-11-24T06:07:31.231 DEBUG:teuthology.orchestra.run.smithi091:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:07:31.233 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:07:32.700 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-11-24T06:07:32.732 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-24T06:07:32.732 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:32.733 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-11-24T06:07:32.764 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:32.764 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-11-24T06:07:32.836 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:32.836 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-11-24T06:07:32.906 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:32.906 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-24T06:07:32.976 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:32.977 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-24T06:07:33.046 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:33.046 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-24T06:07:33.117 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:33.117 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-11-24T06:07:33.185 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:33.185 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-24T06:07:33.217 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:33.217 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-11-24T06:07:33.288 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:33.288 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-11-24T06:07:33.359 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:33.359 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-11-24T06:07:33.429 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:33.430 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-11-24T06:07:33.500 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:33.500 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-11-24T06:07:33.569 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:33.569 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-11-24T06:07:33.637 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:33.638 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-24T06:07:33.657 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:33.658 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-11-24T06:07:33.717 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:33.717 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-11-24T06:07:33.735 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:33.735 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-11-24T06:07:33.792 INFO:tasks.ceph:Adding keys to all mons... 2023-11-24T06:07:33.792 DEBUG:teuthology.orchestra.run.smithi091:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-24T06:07:33.794 DEBUG:teuthology.orchestra.run.smithi094:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-24T06:07:33.835 INFO:teuthology.orchestra.run.smithi091.stdout:[mgr.y] 2023-11-24T06:07:33.835 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCHPWBlDJgdKRAApNIX943pvbdHdb7Wd1qBQA== 2023-11-24T06:07:33.835 INFO:teuthology.orchestra.run.smithi091.stdout:[mds.a] 2023-11-24T06:07:33.835 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCHPWBl/VoLMRAANt18r5ISLrCCFV/DZRefeg== 2023-11-24T06:07:33.835 INFO:teuthology.orchestra.run.smithi091.stdout:[mds.c] 2023-11-24T06:07:33.835 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCHPWBlOJQ7OBAAAb+YjyyqyYSNZUAWn9d2/g== 2023-11-24T06:07:33.836 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.0] 2023-11-24T06:07:33.836 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCLPWBltHiIJhAA0EAQ1DYnQEPevN1UoqneHw== 2023-11-24T06:07:33.836 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.1] 2023-11-24T06:07:33.836 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCOPWBlOQ+kJRAA2F7plkyuC8qRw6qMwrl2uw== 2023-11-24T06:07:33.836 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.2] 2023-11-24T06:07:33.836 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCRPWBlaUL7JBAAiutjTMqJ4POTynJjCvIvDg== 2023-11-24T06:07:33.836 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.3] 2023-11-24T06:07:33.836 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCUPWBlRsdIJBAAMgD3lTDFH3Wc7X+2ye37AQ== 2023-11-24T06:07:33.836 INFO:teuthology.orchestra.run.smithi091.stdout:[mgr.x] 2023-11-24T06:07:33.836 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCHPWBlg46tLRAALjHRKiAhP1of8tSavqqvxg== 2023-11-24T06:07:33.836 INFO:teuthology.orchestra.run.smithi091.stdout:[mds.b] 2023-11-24T06:07:33.836 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCIPWBl4eC/ARAAEgX+/wVFtx5DZbnOtXk35A== 2023-11-24T06:07:33.837 INFO:teuthology.orchestra.run.smithi091.stdout:[mds.d] 2023-11-24T06:07:33.837 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCIPWBlbm0XCRAAcu/fX5wokIxgHqq4vyVblA== 2023-11-24T06:07:33.837 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.4] 2023-11-24T06:07:33.837 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCYPWBlE7tgMxAA13pOQ96qq/9WR6pJtmzKKw== 2023-11-24T06:07:33.837 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.5] 2023-11-24T06:07:33.837 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCbPWBl6o7oMhAAe++S2VE9es9jNzRRg/s0zg== 2023-11-24T06:07:33.837 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.6] 2023-11-24T06:07:33.837 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCePWBl4HZeMhAAefE88W6kjeS1ZMh2YzS/uw== 2023-11-24T06:07:33.837 INFO:teuthology.orchestra.run.smithi091.stdout:[osd.7] 2023-11-24T06:07:33.837 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQChPWBlTGD9MRAA0lf1fKc0GVxhCvp+cckQfg== 2023-11-24T06:07:33.837 INFO:teuthology.orchestra.run.smithi091.stdout:[client.0] 2023-11-24T06:07:33.837 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCIPWBlZntSDRAAdyaD3NYZ7aqLp5+7oaQH+Q== 2023-11-24T06:07:33.837 INFO:teuthology.orchestra.run.smithi091.stdout:[client.2] 2023-11-24T06:07:33.838 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCIPWBlGCICExAA3BvhkJYyZ0SbJLg/PZAjpA== 2023-11-24T06:07:33.838 INFO:teuthology.orchestra.run.smithi091.stdout:[client.1] 2023-11-24T06:07:33.838 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCIPWBltflUFhAAQm7p6/tZmL9YhVNjtakX6g== 2023-11-24T06:07:33.838 INFO:teuthology.orchestra.run.smithi091.stdout:[client.3] 2023-11-24T06:07:33.838 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCIPWBlSIEHHBAAYaIDPZgQUJqKQqGcApp93Q== 2023-11-24T06:07:33.858 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.y] 2023-11-24T06:07:33.858 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCHPWBlDJgdKRAApNIX943pvbdHdb7Wd1qBQA== 2023-11-24T06:07:33.858 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.a] 2023-11-24T06:07:33.858 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCHPWBl/VoLMRAANt18r5ISLrCCFV/DZRefeg== 2023-11-24T06:07:33.858 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.c] 2023-11-24T06:07:33.859 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCHPWBlOJQ7OBAAAb+YjyyqyYSNZUAWn9d2/g== 2023-11-24T06:07:33.859 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.0] 2023-11-24T06:07:33.859 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCLPWBltHiIJhAA0EAQ1DYnQEPevN1UoqneHw== 2023-11-24T06:07:33.859 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.1] 2023-11-24T06:07:33.859 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCOPWBlOQ+kJRAA2F7plkyuC8qRw6qMwrl2uw== 2023-11-24T06:07:33.859 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.2] 2023-11-24T06:07:33.859 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCRPWBlaUL7JBAAiutjTMqJ4POTynJjCvIvDg== 2023-11-24T06:07:33.859 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.3] 2023-11-24T06:07:33.859 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCUPWBlRsdIJBAAMgD3lTDFH3Wc7X+2ye37AQ== 2023-11-24T06:07:33.859 INFO:teuthology.orchestra.run.smithi094.stdout:[mgr.x] 2023-11-24T06:07:33.859 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCHPWBlg46tLRAALjHRKiAhP1of8tSavqqvxg== 2023-11-24T06:07:33.859 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.b] 2023-11-24T06:07:33.860 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCIPWBl4eC/ARAAEgX+/wVFtx5DZbnOtXk35A== 2023-11-24T06:07:33.860 INFO:teuthology.orchestra.run.smithi094.stdout:[mds.d] 2023-11-24T06:07:33.860 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCIPWBlbm0XCRAAcu/fX5wokIxgHqq4vyVblA== 2023-11-24T06:07:33.860 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.4] 2023-11-24T06:07:33.860 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCYPWBlE7tgMxAA13pOQ96qq/9WR6pJtmzKKw== 2023-11-24T06:07:33.860 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.5] 2023-11-24T06:07:33.860 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCbPWBl6o7oMhAAe++S2VE9es9jNzRRg/s0zg== 2023-11-24T06:07:33.860 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.6] 2023-11-24T06:07:33.860 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCePWBl4HZeMhAAefE88W6kjeS1ZMh2YzS/uw== 2023-11-24T06:07:33.860 INFO:teuthology.orchestra.run.smithi094.stdout:[osd.7] 2023-11-24T06:07:33.860 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQChPWBlTGD9MRAA0lf1fKc0GVxhCvp+cckQfg== 2023-11-24T06:07:33.860 INFO:teuthology.orchestra.run.smithi094.stdout:[client.0] 2023-11-24T06:07:33.861 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCIPWBlZntSDRAAdyaD3NYZ7aqLp5+7oaQH+Q== 2023-11-24T06:07:33.861 INFO:teuthology.orchestra.run.smithi094.stdout:[client.2] 2023-11-24T06:07:33.861 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCIPWBlGCICExAA3BvhkJYyZ0SbJLg/PZAjpA== 2023-11-24T06:07:33.861 INFO:teuthology.orchestra.run.smithi094.stdout:[client.1] 2023-11-24T06:07:33.861 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCIPWBltflUFhAAQm7p6/tZmL9YhVNjtakX6g== 2023-11-24T06:07:33.861 INFO:teuthology.orchestra.run.smithi094.stdout:[client.3] 2023-11-24T06:07:33.861 INFO:teuthology.orchestra.run.smithi094.stdout: key = AQCIPWBlSIEHHBAAYaIDPZgQUJqKQqGcApp93Q== 2023-11-24T06:07:33.862 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-24T06:07:33.881 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-24T06:07:33.956 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T06:07:33.958 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T06:07:34.016 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T06:07:34.019 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T06:07:34.074 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:07:34.077 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:07:34.130 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:07:34.132 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:07:34.185 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:07:34.187 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:07:34.240 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:07:34.242 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:07:34.295 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-24T06:07:34.297 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-24T06:07:34.350 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T06:07:34.352 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T06:07:34.408 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T06:07:34.410 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-11-24T06:07:34.466 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:07:34.469 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:07:34.522 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:07:34.524 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:07:34.577 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:07:34.579 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:07:34.633 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:07:34.635 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-24T06:07:34.688 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T06:07:34.690 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T06:07:34.746 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T06:07:34.749 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T06:07:34.805 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T06:07:34.807 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T06:07:34.863 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T06:07:34.865 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-24T06:07:34.920 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-24T06:07:34.921 DEBUG:teuthology.orchestra.run.smithi091:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-24T06:07:34.947 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-24T06:07:35.043 DEBUG:teuthology.orchestra.run.smithi091:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-24T06:07:35.069 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-11-24T06:07:35.095 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-24T06:07:35.193 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-11-24T06:07:35.219 DEBUG:teuthology.orchestra.run.smithi094:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-11-24T06:07:35.285 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-24T06:07:35.382 DEBUG:teuthology.orchestra.run.smithi094:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-11-24T06:07:35.408 DEBUG:teuthology.orchestra.run.smithi091:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:07:35.410 DEBUG:teuthology.orchestra.run.smithi094:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-24T06:07:35.464 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-24T06:07:35.464 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-24T06:07:35.464 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-24T06:07:35.468 INFO:tasks.ceph.mon.a:Started 2023-11-24T06:07:35.468 INFO:tasks.ceph.mon.b:Restarting daemon 2023-11-24T06:07:35.468 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-11-24T06:07:35.506 INFO:tasks.ceph.mon.b:Started 2023-11-24T06:07:35.506 INFO:tasks.ceph.mon.c:Restarting daemon 2023-11-24T06:07:35.506 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-11-24T06:07:35.508 INFO:tasks.ceph.mon.c:Started 2023-11-24T06:07:35.508 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-24T06:07:35.508 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-11-24T06:07:35.508 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-11-24T06:07:35.510 INFO:tasks.ceph.mgr.y:Started 2023-11-24T06:07:35.510 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-24T06:07:35.510 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-24T06:07:35.512 INFO:tasks.ceph.mgr.x:Started 2023-11-24T06:07:35.512 INFO:tasks.ceph:Setting crush tunables to default 2023-11-24T06:07:35.512 DEBUG:teuthology.orchestra.run.smithi091:> sudo ceph --cluster ceph osd crush tunables default 2023-11-24T06:07:35.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.630+0000 7f7564197700 1 Processor -- start 2023-11-24T06:07:35.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.630+0000 7f7564197700 1 -- start start 2023-11-24T06:07:35.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7564197700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c06b620 0x7f755c0b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:35.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7564197700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f755c06ee60 0x7f755c0b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:35.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7564197700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c071f50 con 0x7f755c0b5ab0 2023-11-24T06:07:35.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7564197700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f755c072230 con 0x7f755c06ee60 2023-11-24T06:07:35.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7564197700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755c072510 con 0x7f755c06b620 2023-11-24T06:07:35.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7561732700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f755c0b5ab0 0x7f755c161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:51968/0 (socket says 172.21.15.91:51968) 2023-11-24T06:07:35.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7561732700 1 -- 172.21.15.91:0/3720019551 learned_addr learned my addr 172.21.15.91:0/3720019551 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:07:35.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7562734700 1 --2- 172.21.15.91:0/3720019551 >> v2:172.21.15.94:3300/0 conn(0x7f755c06ee60 0x7f755c0b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:35.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7561f33700 1 --2- 172.21.15.91:0/3720019551 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c06b620 0x7f755c0b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:35.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7561f33700 1 -- 172.21.15.91:0/3720019551 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c06b620 msgr2=0x7f755c0b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-11-24T06:07:35.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7561f33700 1 -- 172.21.15.91:0/3720019551 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c06b620 msgr2=0x7f755c0b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T06:07:35.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7561f33700 1 --2- 172.21.15.91:0/3720019551 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c06b620 0x7f755c0b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T06:07:35.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7562734700 1 -- 172.21.15.91:0/3720019551 >> v2:172.21.15.94:3300/0 conn(0x7f755c06ee60 msgr2=0x7f755c0b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-11-24T06:07:35.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7561f33700 1 --2- 172.21.15.91:0/3720019551 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c06b620 0x7f755c0b3a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-24T06:07:35.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7562734700 1 -- 172.21.15.91:0/3720019551 >> v2:172.21.15.94:3300/0 conn(0x7f755c06ee60 msgr2=0x7f755c0b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-11-24T06:07:35.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7562734700 1 --2- 172.21.15.91:0/3720019551 >> v2:172.21.15.94:3300/0 conn(0x7f755c06ee60 0x7f755c0b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-24T06:07:35.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.631+0000 7f7562734700 1 --2- 172.21.15.91:0/3720019551 >> v2:172.21.15.94:3300/0 conn(0x7f755c06ee60 0x7f755c0b3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-24T06:07:35.820 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.816+0000 7f7560f31700 1 -- 172.21.15.91:0/3720019551 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086890774 0 0) 0x7f755c071f50 con 0x7f755c0b5ab0 2023-11-24T06:07:35.820 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.816+0000 7f7560f31700 1 -- 172.21.15.91:0/3720019551 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75480035b0 con 0x7f755c0b5ab0 2023-11-24T06:07:35.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7560f31700 1 -- 172.21.15.91:0/3720019551 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1386397235 0 0) 0x7f75480035b0 con 0x7f755c0b5ab0 2023-11-24T06:07:35.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7560f31700 1 -- 172.21.15.91:0/3720019551 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c06b620 msgr2=0x7f755c0b3a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-24T06:07:35.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7560f31700 1 --2- 172.21.15.91:0/3720019551 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c06b620 0x7f755c0b3a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:35.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7560f31700 1 -- 172.21.15.91:0/3720019551 >> v2:172.21.15.94:3300/0 conn(0x7f755c06ee60 msgr2=0x7f755c0b3f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-24T06:07:35.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7560f31700 1 --2- 172.21.15.91:0/3720019551 >> v2:172.21.15.94:3300/0 conn(0x7f755c06ee60 0x7f755c0b3f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:35.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7560f31700 1 -- 172.21.15.91:0/3720019551 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c0727f0 con 0x7f755c0b5ab0 2023-11-24T06:07:35.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7560f31700 1 -- 172.21.15.91:0/3720019551 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f7558001670 con 0x7f755c0b5ab0 2023-11-24T06:07:35.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7560f31700 1 -- 172.21.15.91:0/3720019551 >> v1:172.21.15.91:6789/0 conn(0x7f755c0b5ab0 legacy=0x7f755c161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:35.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7560f31700 1 --2- 172.21.15.91:0/3720019551 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c06b620 0x7f7548003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:35.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7560f31700 1 --2- 172.21.15.91:0/3720019551 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7548004030 0x7f7548004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:35.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7560f31700 1 --2- 172.21.15.91:0/3720019551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7548004d90 0x7f7548005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:35.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7560f31700 1 -- 172.21.15.91:0/3720019551 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c072230 con 0x7f7548004030 2023-11-24T06:07:35.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7560f31700 1 -- 172.21.15.91:0/3720019551 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c072510 con 0x7f7548004d90 2023-11-24T06:07:35.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7560f31700 1 -- 172.21.15.91:0/3720019551 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754800bb10 con 0x7f755c06b620 2023-11-24T06:07:35.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7562734700 1 --2- 172.21.15.91:0/3720019551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7548004d90 0x7f7548005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:35.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7561732700 1 --2- 172.21.15.91:0/3720019551 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7548004030 0x7f7548004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:35.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7561f33700 1 --2- 172.21.15.91:0/3720019551 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c06b620 0x7f7548003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:35.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7561732700 1 -- 172.21.15.91:0/3720019551 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c06b620 msgr2=0x7f7548003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:35.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7561732700 1 --2- 172.21.15.91:0/3720019551 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c06b620 0x7f7548003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:35.824 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7561732700 1 -- 172.21.15.91:0/3720019551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7548004d90 msgr2=0x7f7548005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:35.824 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7561732700 1 --2- 172.21.15.91:0/3720019551 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7548004d90 0x7f7548005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:35.824 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7561732700 1 -- 172.21.15.91:0/3720019551 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75480035b0 con 0x7f7548004030 2023-11-24T06:07:35.824 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.817+0000 7f7561732700 1 --2- 172.21.15.91:0/3720019551 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7548004030 0x7f7548004850 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f755c0b5f10 tx=0x7f7548005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b075b72f42bec6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:35.825 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.819+0000 7f7560f31700 1 -- 172.21.15.91:0/3720019551 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558007c60 con 0x7f7548004030 2023-11-24T06:07:35.825 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.819+0000 7f7560f31700 1 -- 172.21.15.91:0/3720019551 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755800ce80 con 0x7f7548004030 2023-11-24T06:07:35.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.819+0000 7f7560f31700 1 -- 172.21.15.91:0/3720019551 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7558005ba0 con 0x7f7548004030 2023-11-24T06:07:35.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.820+0000 7f7564197700 1 -- 172.21.15.91:0/3720019551 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7548004030 msgr2=0x7f7548004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:35.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.820+0000 7f7564197700 1 --2- 172.21.15.91:0/3720019551 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7548004030 0x7f7548004850 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f755c0b5f10 tx=0x7f7548005880 comp rx=0 tx=0).stop 2023-11-24T06:07:35.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.820+0000 7f7562734700 1 -- 172.21.15.91:0/3720019551 reap_dead start 2023-11-24T06:07:35.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.820+0000 7f7564197700 1 -- 172.21.15.91:0/3720019551 shutdown_connections 2023-11-24T06:07:35.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.820+0000 7f7564197700 1 -- 172.21.15.91:0/3720019551 >> 172.21.15.91:0/3720019551 conn(0x7f755c0b2740 msgr2=0x7f755c0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:35.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.820+0000 7f7564197700 1 -- 172.21.15.91:0/3720019551 shutdown_connections 2023-11-24T06:07:35.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.820+0000 7f7564197700 1 -- 172.21.15.91:0/3720019551 wait complete. 2023-11-24T06:07:35.827 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.821+0000 7f7564197700 1 Processor -- start 2023-11-24T06:07:35.827 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.821+0000 7f7564197700 1 -- start start 2023-11-24T06:07:35.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7564197700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7548004030 0x7f755c1e7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:35.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7564197700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f755c0c59b0 0x7f755c142d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:35.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7564197700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c1f42c0 0x7f755c1e4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:35.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7564197700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c1e7a60 con 0x7f7548004030 2023-11-24T06:07:35.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7564197700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c1e7d40 con 0x7f755c0c59b0 2023-11-24T06:07:35.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7564197700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755c1e8020 con 0x7f755c1f42c0 2023-11-24T06:07:35.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7561f33700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7548004030 0x7f755c1e7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:35.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7561f33700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7548004030 0x7f755c1e7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57066/0 (socket says 172.21.15.91:57066) 2023-11-24T06:07:35.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7561f33700 1 -- 172.21.15.91:0/2969447097 learned_addr learned my addr 172.21.15.91:0/2969447097 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:35.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7561732700 1 --2- 172.21.15.91:0/2969447097 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f755c0c59b0 0x7f755c142d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:35.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7562734700 1 --2- 172.21.15.91:0/2969447097 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c1f42c0 0x7f755c1e4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:35.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7561f33700 1 -- 172.21.15.91:0/2969447097 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c1f42c0 msgr2=0x7f755c1e4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:35.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7561f33700 1 --2- 172.21.15.91:0/2969447097 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c1f42c0 0x7f755c1e4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:35.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7561f33700 1 -- 172.21.15.91:0/2969447097 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f755c0c59b0 msgr2=0x7f755c142d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:35.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7561f33700 1 --2- 172.21.15.91:0/2969447097 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f755c0c59b0 0x7f755c142d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:35.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7561f33700 1 -- 172.21.15.91:0/2969447097 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c1e8300 con 0x7f7548004030 2023-11-24T06:07:35.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.822+0000 7f7561732700 1 --2- 172.21.15.91:0/2969447097 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f755c0c59b0 0x7f755c142d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:07:35.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.823+0000 7f7561f33700 1 --2- 172.21.15.91:0/2969447097 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7548004030 0x7f755c1e7480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f754c0109f0 tx=0x7f754c010d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:35.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.823+0000 7f75527fc700 1 -- 172.21.15.91:0/2969447097 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c006ea0 con 0x7f7548004030 2023-11-24T06:07:35.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.823+0000 7f75527fc700 1 -- 172.21.15.91:0/2969447097 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754c00ea70 con 0x7f7548004030 2023-11-24T06:07:35.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.823+0000 7f7564197700 1 -- 172.21.15.91:0/2969447097 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f755c1e85e0 con 0x7f7548004030 2023-11-24T06:07:35.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.823+0000 7f75527fc700 1 -- 172.21.15.91:0/2969447097 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c009730 con 0x7f7548004030 2023-11-24T06:07:35.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.823+0000 7f7564197700 1 -- 172.21.15.91:0/2969447097 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f755c0c5e30 con 0x7f7548004030 2023-11-24T06:07:35.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.823+0000 7f75527fc700 1 -- 172.21.15.91:0/2969447097 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f754c015470 con 0x7f7548004030 2023-11-24T06:07:35.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.823+0000 7f75527fc700 1 -- 172.21.15.91:0/2969447097 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f754c016420 con 0x7f7548004030 2023-11-24T06:07:35.832 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.824+0000 7f7564197700 1 -- 172.21.15.91:0/2969447097 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755c075480 con 0x7f7548004030 2023-11-24T06:07:35.832 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.826+0000 7f75527fc700 1 -- 172.21.15.91:0/2969447097 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f754c01a020 con 0x7f7548004030 2023-11-24T06:07:35.907 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.903+0000 7f7564197700 1 -- 172.21.15.91:0/2969447097 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f755c0b5540 con 0x7f7548004030 2023-11-24T06:07:35.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.905+0000 7f75527fc700 1 -- 172.21.15.91:0/2969447097 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f754c00f040 con 0x7f7548004030 2023-11-24T06:07:35.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.906+0000 7f7564197700 1 -- 172.21.15.91:0/2969447097 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7548004030 msgr2=0x7f755c1e7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:35.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.906+0000 7f7564197700 1 --2- 172.21.15.91:0/2969447097 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7548004030 0x7f755c1e7480 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f754c0109f0 tx=0x7f754c010d00 comp rx=0 tx=0).stop 2023-11-24T06:07:35.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.906+0000 7f7564197700 1 -- 172.21.15.91:0/2969447097 shutdown_connections 2023-11-24T06:07:35.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.906+0000 7f7564197700 1 --2- 172.21.15.91:0/2969447097 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f755c1f42c0 0x7f755c1e4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:35.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.906+0000 7f7564197700 1 --2- 172.21.15.91:0/2969447097 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7548004030 0x7f755c1e7480 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:35.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.906+0000 7f7564197700 1 --2- 172.21.15.91:0/2969447097 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f755c0c59b0 0x7f755c142d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:35.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.906+0000 7f7564197700 1 -- 172.21.15.91:0/2969447097 >> 172.21.15.91:0/2969447097 conn(0x7f755c0b2740 msgr2=0x7f755c1e9820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:35.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.906+0000 7f7564197700 1 -- 172.21.15.91:0/2969447097 shutdown_connections 2023-11-24T06:07:35.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:35.906+0000 7f7564197700 1 -- 172.21.15.91:0/2969447097 wait complete. 2023-11-24T06:07:35.912 INFO:teuthology.orchestra.run.smithi091.stderr:adjusted tunables profile to default 2023-11-24T06:07:35.921 INFO:tasks.ceph:check_enable_crimson: False 2023-11-24T06:07:35.922 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-24T06:07:35.922 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:35.922 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-24T06:07:35.935 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:35.932+0000 7f17cbd25200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-24T06:07:35.938 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:35.934+0000 7ff0604e1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-24T06:07:35.948 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:35.948 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-24T06:07:36.000 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:35.997+0000 7f17cbd25200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-24T06:07:36.006 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:36.003+0000 7ff0604e1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-24T06:07:36.015 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:36.015 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-24T06:07:36.076 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:36.073+0000 7f17cbd25200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-24T06:07:36.082 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:07:36.082 DEBUG:teuthology.orchestra.run.smithi091:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-11-24T06:07:36.091 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:36.087+0000 7ff0604e1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-24T06:07:36.119 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:36.119 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-11-24T06:07:36.146 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:36.146 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-11-24T06:07:36.157 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:36.154+0000 7f17cbd25200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-24T06:07:36.175 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:36.172+0000 7ff0604e1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-24T06:07:36.182 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:36.182 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-11-24T06:07:36.249 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:07:36.249 DEBUG:teuthology.orchestra.run.smithi094:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-11-24T06:07:36.306 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:36.303+0000 7f17cbd25200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-24T06:07:36.316 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 5296abf7-43c1-435b-87fc-71b8cf881bc5 0 2023-11-24T06:07:36.317 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:36.312+0000 7ff0604e1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-24T06:07:36.478 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.475+0000 7ff5dd7f6700 1 Processor -- start 2023-11-24T06:07:36.478 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.475+0000 7ff5dd7f6700 1 -- start start 2023-11-24T06:07:36.481 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.476+0000 7ff5dd7f6700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff5d806b620 0x7ff5d80b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:36.481 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.476+0000 7ff5dd7f6700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff5d80b5ab0 0x7ff5d80b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:36.481 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.476+0000 7ff5dd7f6700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d8071f50 con 0x7ff5d81478c0 2023-11-24T06:07:36.481 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.476+0000 7ff5dd7f6700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5d8072230 con 0x7ff5d806b620 2023-11-24T06:07:36.481 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.476+0000 7ff5dd7f6700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5d8072510 con 0x7ff5d80b5ab0 2023-11-24T06:07:36.481 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.476+0000 7ff5d67fc700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff5d80b5ab0 0x7ff5d80b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:36.481 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.476+0000 7ff5d67fc700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff5d80b5ab0 0x7ff5d80b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52184/0 (socket says 172.21.15.94:52184) 2023-11-24T06:07:36.481 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.476+0000 7ff5d67fc700 1 -- 172.21.15.94:0/53054432 learned_addr learned my addr 172.21.15.94:0/53054432 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-11-24T06:07:36.481 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.476+0000 7ff5d6ffd700 1 --2- 172.21.15.94:0/53054432 >> v2:172.21.15.91:3300/0 conn(0x7ff5d806b620 0x7ff5d80b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:36.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5d67fc700 1 -- 172.21.15.94:0/53054432 >> v2:172.21.15.91:3300/0 conn(0x7ff5d806b620 msgr2=0x7ff5d80b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:36.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5d67fc700 1 --2- 172.21.15.94:0/53054432 >> v2:172.21.15.91:3300/0 conn(0x7ff5d806b620 0x7ff5d80b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5d67fc700 1 -- 172.21.15.94:0/53054432 >> v1:172.21.15.91:6789/0 conn(0x7ff5d81478c0 legacy=0x7ff5d8161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:36.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5d5ffb700 1 -- 172.21.15.94:0/53054432 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638394301 0 0) 0x7ff5d8071f50 con 0x7ff5d81478c0 2023-11-24T06:07:36.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5d67fc700 1 -- 172.21.15.94:0/53054432 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d80727f0 con 0x7ff5d80b5ab0 2023-11-24T06:07:36.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5d67fc700 1 --2- 172.21.15.94:0/53054432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d80b5ab0 0x7ff5d80b3f80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff5d806d6f0 tx=0x7ff5cc009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6737df1d6d8064b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:36.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5d5ffb700 1 -- 172.21.15.94:0/53054432 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc011830 con 0x7ff5d80b5ab0 2023-11-24T06:07:36.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5d5ffb700 1 -- 172.21.15.94:0/53054432 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5cc011990 con 0x7ff5d80b5ab0 2023-11-24T06:07:36.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5d5ffb700 1 -- 172.21.15.94:0/53054432 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc011d20 con 0x7ff5d80b5ab0 2023-11-24T06:07:36.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5dd7f6700 1 -- 172.21.15.94:0/53054432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d80b5ab0 msgr2=0x7ff5d80b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:36.482 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5dd7f6700 1 --2- 172.21.15.94:0/53054432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d80b5ab0 0x7ff5d80b3f80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff5d806d6f0 tx=0x7ff5cc009bd0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5dd7f6700 1 -- 172.21.15.94:0/53054432 shutdown_connections 2023-11-24T06:07:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5dd7f6700 1 --2- 172.21.15.94:0/53054432 >> v2:172.21.15.91:3300/0 conn(0x7ff5d806b620 0x7ff5d80b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5dd7f6700 1 --2- 172.21.15.94:0/53054432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d80b5ab0 0x7ff5d80b3f80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5dd7f6700 1 -- 172.21.15.94:0/53054432 >> 172.21.15.94:0/53054432 conn(0x7ff5d80b2740 msgr2=0x7ff5d80b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5dd7f6700 1 -- 172.21.15.94:0/53054432 shutdown_connections 2023-11-24T06:07:36.483 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.477+0000 7ff5dd7f6700 1 -- 172.21.15.94:0/53054432 wait complete. 2023-11-24T06:07:36.485 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.479+0000 7ff5dd7f6700 1 Processor -- start 2023-11-24T06:07:36.485 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.479+0000 7ff5dd7f6700 1 -- start start 2023-11-24T06:07:36.485 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.479+0000 7ff5dd7f6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d806b620 0x7ff5d813da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:36.485 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.479+0000 7ff5dd7f6700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff5d80b5ab0 0x7ff5d80c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:36.485 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.479+0000 7ff5dd7f6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff5d81478c0 0x7ff5d806e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:36.486 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.479+0000 7ff5dd7f6700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d8140130 con 0x7ff5d80b5ab0 2023-11-24T06:07:36.486 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.479+0000 7ff5dd7f6700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d813f580 con 0x7ff5d806b620 2023-11-24T06:07:36.486 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.479+0000 7ff5dd7f6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d813df50 con 0x7ff5d81478c0 2023-11-24T06:07:36.486 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5d6ffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d806b620 0x7ff5d813da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:36.486 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5d6ffd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d806b620 0x7ff5d813da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52188/0 (socket says 172.21.15.94:52188) 2023-11-24T06:07:36.486 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5d6ffd700 1 -- 172.21.15.94:0/3546789465 learned_addr learned my addr 172.21.15.94:0/3546789465 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-11-24T06:07:36.486 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5d67fc700 1 --2- 172.21.15.94:0/3546789465 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff5d80b5ab0 0x7ff5d80c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:36.486 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5d77fe700 1 --2- 172.21.15.94:0/3546789465 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff5d81478c0 0x7ff5d806e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:36.486 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5d6ffd700 1 -- 172.21.15.94:0/3546789465 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff5d81478c0 msgr2=0x7ff5d806e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:36.486 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5d6ffd700 1 --2- 172.21.15.94:0/3546789465 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff5d81478c0 0x7ff5d806e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.486 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5d6ffd700 1 -- 172.21.15.94:0/3546789465 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff5d80b5ab0 msgr2=0x7ff5d80c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:36.486 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5d6ffd700 1 --2- 172.21.15.94:0/3546789465 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff5d80b5ab0 0x7ff5d80c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.487 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5d6ffd700 1 -- 172.21.15.94:0/3546789465 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5cc009550 con 0x7ff5d806b620 2023-11-24T06:07:36.487 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5d6ffd700 1 --2- 172.21.15.94:0/3546789465 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d806b620 0x7ff5d813da10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff5c800e9b0 tx=0x7ff5c800ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:36.487 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5bf7fe700 1 -- 172.21.15.94:0/3546789465 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c8009710 con 0x7ff5d806b620 2023-11-24T06:07:36.487 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5bf7fe700 1 -- 172.21.15.94:0/3546789465 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c800ee50 con 0x7ff5d806b620 2023-11-24T06:07:36.487 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5dd7f6700 1 -- 172.21.15.94:0/3546789465 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d813f790 con 0x7ff5d806b620 2023-11-24T06:07:36.487 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5bf7fe700 1 -- 172.21.15.94:0/3546789465 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c800ca50 con 0x7ff5d806b620 2023-11-24T06:07:36.487 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5dd7f6700 1 -- 172.21.15.94:0/3546789465 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d813fd50 con 0x7ff5d806b620 2023-11-24T06:07:36.487 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5bf7fe700 1 -- 172.21.15.94:0/3546789465 <== mon.1 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff5c8005a20 con 0x7ff5d806b620 2023-11-24T06:07:36.487 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.480+0000 7ff5bf7fe700 1 -- 172.21.15.94:0/3546789465 <== mon.1 v2:172.21.15.94:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff5c80137b0 con 0x7ff5d806b620 2023-11-24T06:07:36.487 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.481+0000 7ff5dd7f6700 1 -- 172.21.15.94:0/3546789465 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5d8075480 con 0x7ff5d806b620 2023-11-24T06:07:36.488 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.483+0000 7ff5bf7fe700 1 -- 172.21.15.94:0/3546789465 <== mon.1 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ff5c8021060 con 0x7ff5d806b620 2023-11-24T06:07:36.571 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.567+0000 7ff5dd7f6700 1 -- 172.21.15.94:0/3546789465 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5296abf7-43c1-435b-87fc-71b8cf881bc5", "id": 0} v 0) v1 -- 0x7ff5d806bd40 con 0x7ff5d806b620 2023-11-24T06:07:36.574 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.570+0000 7ff5bf7fe700 1 -- 172.21.15.94:0/3546789465 <== mon.1 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5296abf7-43c1-435b-87fc-71b8cf881bc5", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff5c8017070 con 0x7ff5d806b620 2023-11-24T06:07:36.574 INFO:teuthology.orchestra.run.smithi094.stdout:0 2023-11-24T06:07:36.575 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.571+0000 7ff5dd7f6700 1 -- 172.21.15.94:0/3546789465 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d806b620 msgr2=0x7ff5d813da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:36.575 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.571+0000 7ff5dd7f6700 1 --2- 172.21.15.94:0/3546789465 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d806b620 0x7ff5d813da10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff5c800e9b0 tx=0x7ff5c800ecc0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.575 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.571+0000 7ff5dd7f6700 1 -- 172.21.15.94:0/3546789465 shutdown_connections 2023-11-24T06:07:36.576 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.571+0000 7ff5dd7f6700 1 --2- 172.21.15.94:0/3546789465 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5d806b620 0x7ff5d813da10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.576 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.571+0000 7ff5dd7f6700 1 --2- 172.21.15.94:0/3546789465 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff5d81478c0 0x7ff5d806e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.576 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.571+0000 7ff5dd7f6700 1 --2- 172.21.15.94:0/3546789465 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff5d80b5ab0 0x7ff5d80c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.576 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.571+0000 7ff5dd7f6700 1 -- 172.21.15.94:0/3546789465 >> 172.21.15.94:0/3546789465 conn(0x7ff5d80b2740 msgr2=0x7ff5d80b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:36.576 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.571+0000 7ff5dd7f6700 1 -- 172.21.15.94:0/3546789465 shutdown_connections 2023-11-24T06:07:36.576 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.571+0000 7ff5dd7f6700 1 -- 172.21.15.94:0/3546789465 wait complete. 2023-11-24T06:07:36.586 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new fc021485-f189-4918-b6e8-bef2e064a46b 1 2023-11-24T06:07:36.706 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:36.702+0000 7f17cbd25200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-24T06:07:36.706 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.703+0000 7f6befacd700 1 Processor -- start 2023-11-24T06:07:36.707 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.703+0000 7f6befacd700 1 -- start start 2023-11-24T06:07:36.709 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.704+0000 7f6befacd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f6be815bde0 0x7f6be815c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:36.709 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.704+0000 7f6befacd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6be80ce5c0 0x7f6be80cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:36.709 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.704+0000 7f6befacd700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be80d0a90 con 0x7f6be80cf2b0 2023-11-24T06:07:36.709 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.704+0000 7f6befacd700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be80cefb0 con 0x7f6be815bde0 2023-11-24T06:07:36.709 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.704+0000 7f6befacd700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be806ee10 con 0x7f6be80ce5c0 2023-11-24T06:07:36.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.704+0000 7f6bee06a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6be80ce5c0 0x7f6be80cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:36.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.704+0000 7f6bed068700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6be80cf2b0 0x7f6be80d01b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33356/0 (socket says 172.21.15.94:33356) 2023-11-24T06:07:36.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.704+0000 7f6bed068700 1 -- 172.21.15.94:0/594470748 learned_addr learned my addr 172.21.15.94:0/594470748 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-11-24T06:07:36.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.705+0000 7f6bed869700 1 --2- 172.21.15.94:0/594470748 >> v2:172.21.15.91:3300/0 conn(0x7f6be815bde0 0x7f6be815c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:36.710 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.705+0000 7f6bec867700 1 -- 172.21.15.94:0/594470748 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631423416 0 0) 0x7f6be80d0a90 con 0x7f6be80cf2b0 2023-11-24T06:07:36.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.705+0000 7f6bec867700 1 -- 172.21.15.94:0/594470748 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bd40035b0 con 0x7f6be80cf2b0 2023-11-24T06:07:36.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.705+0000 7f6bee06a700 1 -- 172.21.15.94:0/594470748 >> v2:172.21.15.91:3300/0 conn(0x7f6be815bde0 msgr2=0x7f6be815c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:36.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.705+0000 7f6bee06a700 1 --2- 172.21.15.94:0/594470748 >> v2:172.21.15.91:3300/0 conn(0x7f6be815bde0 0x7f6be815c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.705+0000 7f6bee06a700 1 -- 172.21.15.94:0/594470748 >> v1:172.21.15.94:6789/0 conn(0x7f6be80cf2b0 legacy=0x7f6be80d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:36.711 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.705+0000 7f6bee06a700 1 -- 172.21.15.94:0/594470748 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be806f0f0 con 0x7f6be80ce5c0 2023-11-24T06:07:36.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.705+0000 7f6bee06a700 1 --2- 172.21.15.94:0/594470748 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6be80ce5c0 0x7f6be80cea70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009d90 tx=0x7f6be000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43e868ff0eea22f7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:36.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.705+0000 7f6bec867700 1 -- 172.21.15.94:0/594470748 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be000e070 con 0x7f6be80ce5c0 2023-11-24T06:07:36.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.705+0000 7f6bec867700 1 -- 172.21.15.94:0/594470748 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be000bb20 con 0x7f6be80ce5c0 2023-11-24T06:07:36.712 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.705+0000 7f6bec867700 1 -- 172.21.15.94:0/594470748 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be000be90 con 0x7f6be80ce5c0 2023-11-24T06:07:36.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.706+0000 7f6befacd700 1 -- 172.21.15.94:0/594470748 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6be80ce5c0 msgr2=0x7f6be80cea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:36.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.706+0000 7f6befacd700 1 --2- 172.21.15.94:0/594470748 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6be80ce5c0 0x7f6be80cea70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6be0009d90 tx=0x7f6be000b330 comp rx=0 tx=0).stop 2023-11-24T06:07:36.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.706+0000 7f6befacd700 1 -- 172.21.15.94:0/594470748 shutdown_connections 2023-11-24T06:07:36.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.706+0000 7f6befacd700 1 --2- 172.21.15.94:0/594470748 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6be80ce5c0 0x7f6be80cea70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.713 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.706+0000 7f6befacd700 1 --2- 172.21.15.94:0/594470748 >> v2:172.21.15.91:3300/0 conn(0x7f6be815bde0 0x7f6be815c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.714 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.706+0000 7f6befacd700 1 -- 172.21.15.94:0/594470748 >> 172.21.15.94:0/594470748 conn(0x7f6be80b3720 msgr2=0x7f6be80b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:36.714 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.706+0000 7f6befacd700 1 -- 172.21.15.94:0/594470748 shutdown_connections 2023-11-24T06:07:36.714 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.706+0000 7f6befacd700 1 -- 172.21.15.94:0/594470748 wait complete. 2023-11-24T06:07:36.715 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:36.705+0000 7ff0604e1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-24T06:07:36.717 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.707+0000 7f6befacd700 1 Processor -- start 2023-11-24T06:07:36.717 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.707+0000 7f6befacd700 1 -- start start 2023-11-24T06:07:36.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6befacd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6be80ce5c0 0x7f6be8153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:36.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6befacd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be80cf2b0 0x7f6be80c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:36.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6befacd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6be815bde0 0x7f6be806cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:36.718 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6befacd700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be8153750 con 0x7f6be815bde0 2023-11-24T06:07:36.719 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6befacd700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be80b4b80 con 0x7f6be80cf2b0 2023-11-24T06:07:36.719 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6befacd700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be80b4e60 con 0x7f6be80ce5c0 2023-11-24T06:07:36.719 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6bed869700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6be80ce5c0 0x7f6be8153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:36.719 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6bed068700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be80cf2b0 0x7f6be80c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:36.720 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6bed869700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6be80ce5c0 0x7f6be8153210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:50816/0 (socket says 172.21.15.94:50816) 2023-11-24T06:07:36.720 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6bed869700 1 -- 172.21.15.94:0/3316639192 learned_addr learned my addr 172.21.15.94:0/3316639192 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-11-24T06:07:36.720 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6bee06a700 1 --2- 172.21.15.94:0/3316639192 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6be815bde0 0x7f6be806cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:36.720 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6bed869700 1 -- 172.21.15.94:0/3316639192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be80cf2b0 msgr2=0x7f6be80c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:36.720 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6bed869700 1 --2- 172.21.15.94:0/3316639192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be80cf2b0 0x7f6be80c5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.720 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6bed869700 1 -- 172.21.15.94:0/3316639192 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6be815bde0 msgr2=0x7f6be806cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:36.720 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6bed869700 1 --2- 172.21.15.94:0/3316639192 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6be815bde0 0x7f6be806cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.720 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6bed869700 1 -- 172.21.15.94:0/3316639192 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be0009a70 con 0x7f6be80ce5c0 2023-11-24T06:07:36.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6bed068700 1 --2- 172.21.15.94:0/3316639192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be80cf2b0 0x7f6be80c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:07:36.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6bed869700 1 --2- 172.21.15.94:0/3316639192 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6be80ce5c0 0x7f6be8153210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6be400e870 tx=0x7f6be400eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:36.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6bde7fc700 1 -- 172.21.15.94:0/3316639192 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be4009590 con 0x7f6be80ce5c0 2023-11-24T06:07:36.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6bde7fc700 1 -- 172.21.15.94:0/3316639192 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be4013680 con 0x7f6be80ce5c0 2023-11-24T06:07:36.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6bde7fc700 1 -- 172.21.15.94:0/3316639192 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be40139f0 con 0x7f6be80ce5c0 2023-11-24T06:07:36.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.708+0000 7f6befacd700 1 -- 172.21.15.94:0/3316639192 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be80b5140 con 0x7f6be80ce5c0 2023-11-24T06:07:36.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.709+0000 7f6befacd700 1 -- 172.21.15.94:0/3316639192 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be80b5700 con 0x7f6be80ce5c0 2023-11-24T06:07:36.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.709+0000 7f6bde7fc700 1 -- 172.21.15.94:0/3316639192 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6be401c450 con 0x7f6be80ce5c0 2023-11-24T06:07:36.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.709+0000 7f6bde7fc700 1 -- 172.21.15.94:0/3316639192 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f6be4012900 con 0x7f6be80ce5c0 2023-11-24T06:07:36.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.710+0000 7f6befacd700 1 -- 172.21.15.94:0/3316639192 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be80b5a70 con 0x7f6be80ce5c0 2023-11-24T06:07:36.721 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.712+0000 7f6bde7fc700 1 -- 172.21.15.94:0/3316639192 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f6be4021060 con 0x7f6be80ce5c0 2023-11-24T06:07:36.801 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.797+0000 7f6befacd700 1 -- 172.21.15.94:0/3316639192 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd new", "uuid": "fc021485-f189-4918-b6e8-bef2e064a46b", "id": 1} v 0) v1 -- 0x7f6be8075480 con 0x7f6be80ce5c0 2023-11-24T06:07:36.804 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.800+0000 7f6bde7fc700 1 -- 172.21.15.94:0/3316639192 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fc021485-f189-4918-b6e8-bef2e064a46b", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6be4017070 con 0x7f6be80ce5c0 2023-11-24T06:07:36.804 INFO:teuthology.orchestra.run.smithi094.stdout:1 2023-11-24T06:07:36.805 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.801+0000 7f6befacd700 1 -- 172.21.15.94:0/3316639192 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6be80ce5c0 msgr2=0x7f6be8153210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:36.805 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.801+0000 7f6befacd700 1 --2- 172.21.15.94:0/3316639192 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6be80ce5c0 0x7f6be8153210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6be400e870 tx=0x7f6be400eb80 comp rx=0 tx=0).stop 2023-11-24T06:07:36.805 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.801+0000 7f6befacd700 1 -- 172.21.15.94:0/3316639192 shutdown_connections 2023-11-24T06:07:36.805 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.801+0000 7f6befacd700 1 --2- 172.21.15.94:0/3316639192 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6be815bde0 0x7f6be806cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.805 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.801+0000 7f6befacd700 1 --2- 172.21.15.94:0/3316639192 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6be80ce5c0 0x7f6be8153210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.805 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.801+0000 7f6befacd700 1 --2- 172.21.15.94:0/3316639192 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6be80cf2b0 0x7f6be80c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.805 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.801+0000 7f6befacd700 1 -- 172.21.15.94:0/3316639192 >> 172.21.15.94:0/3316639192 conn(0x7f6be80b3720 msgr2=0x7f6be815b120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:36.806 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.802+0000 7f6befacd700 1 -- 172.21.15.94:0/3316639192 shutdown_connections 2023-11-24T06:07:36.806 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.802+0000 7f6befacd700 1 -- 172.21.15.94:0/3316639192 wait complete. 2023-11-24T06:07:36.816 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 6938ed71-88b6-47e4-9c1d-fbc2856d74f9 2 2023-11-24T06:07:36.828 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:36.825+0000 7f17cbd25200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-24T06:07:36.830 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:36.827+0000 7ff0604e1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-24T06:07:36.928 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:36.924+0000 7ff0604e1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-24T06:07:36.930 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:36.927+0000 7f17cbd25200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-24T06:07:36.936 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.933+0000 7fa6180b0700 1 Processor -- start 2023-11-24T06:07:36.937 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.934+0000 7fa6180b0700 1 -- start start 2023-11-24T06:07:36.940 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.934+0000 7fa6180b0700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fa61015f380 0x7fa610066ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:36.940 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.934+0000 7fa6180b0700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa6100b5eb0 0x7fa61006bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:36.940 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.934+0000 7fa6180b0700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61015f760 con 0x7fa6100cc8e0 2023-11-24T06:07:36.940 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.934+0000 7fa6180b0700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61006b2f0 con 0x7fa61015f380 2023-11-24T06:07:36.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.934+0000 7fa6180b0700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61006b5b0 con 0x7fa6100b5eb0 2023-11-24T06:07:36.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.934+0000 7fa61564b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa6100b5eb0 0x7fa61006bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:36.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.934+0000 7fa61564b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa6100b5eb0 0x7fa61006bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52208/0 (socket says 172.21.15.94:52208) 2023-11-24T06:07:36.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.934+0000 7fa61564b700 1 -- 172.21.15.94:0/4254907290 learned_addr learned my addr 172.21.15.94:0/4254907290 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-11-24T06:07:36.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.935+0000 7fa615e4c700 1 --2- 172.21.15.94:0/4254907290 >> v2:172.21.15.91:3300/0 conn(0x7fa61015f380 0x7fa610066ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:36.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.935+0000 7fa61564b700 1 -- 172.21.15.94:0/4254907290 >> v2:172.21.15.91:3300/0 conn(0x7fa61015f380 msgr2=0x7fa610066ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:36.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.935+0000 7fa61564b700 1 --2- 172.21.15.94:0/4254907290 >> v2:172.21.15.91:3300/0 conn(0x7fa61015f380 0x7fa610066ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.935+0000 7fa61564b700 1 -- 172.21.15.94:0/4254907290 >> v1:172.21.15.91:6789/0 conn(0x7fa6100cc8e0 legacy=0x7fa610072710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:36.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.935+0000 7fa61564b700 1 -- 172.21.15.94:0/4254907290 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61006b890 con 0x7fa6100b5eb0 2023-11-24T06:07:36.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.935+0000 7fa61564b700 1 --2- 172.21.15.94:0/4254907290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6100b5eb0 0x7fa61006bf60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa6100712b0 tx=0x7fa600009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cf2e25d880da3a6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:36.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.935+0000 7fa614e4a700 1 -- 172.21.15.94:0/4254907290 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa600011830 con 0x7fa6100b5eb0 2023-11-24T06:07:36.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.935+0000 7fa614e4a700 1 -- 172.21.15.94:0/4254907290 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa600011990 con 0x7fa6100b5eb0 2023-11-24T06:07:36.941 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.935+0000 7fa614e4a700 1 -- 172.21.15.94:0/4254907290 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa600011d20 con 0x7fa6100b5eb0 2023-11-24T06:07:36.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.936+0000 7fa6180b0700 1 -- 172.21.15.94:0/4254907290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6100b5eb0 msgr2=0x7fa61006bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:36.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.936+0000 7fa6180b0700 1 --2- 172.21.15.94:0/4254907290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6100b5eb0 0x7fa61006bf60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa6100712b0 tx=0x7fa600009bd0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.936+0000 7fa6180b0700 1 -- 172.21.15.94:0/4254907290 shutdown_connections 2023-11-24T06:07:36.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.936+0000 7fa6180b0700 1 --2- 172.21.15.94:0/4254907290 >> v2:172.21.15.91:3300/0 conn(0x7fa61015f380 0x7fa610066ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.936+0000 7fa6180b0700 1 --2- 172.21.15.94:0/4254907290 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6100b5eb0 0x7fa61006bf60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.936+0000 7fa6180b0700 1 -- 172.21.15.94:0/4254907290 >> 172.21.15.94:0/4254907290 conn(0x7fa6100b5070 msgr2=0x7fa6100b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:36.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.936+0000 7fa6180b0700 1 -- 172.21.15.94:0/4254907290 shutdown_connections 2023-11-24T06:07:36.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.936+0000 7fa6180b0700 1 -- 172.21.15.94:0/4254907290 wait complete. 2023-11-24T06:07:36.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.937+0000 7fa6180b0700 1 Processor -- start 2023-11-24T06:07:36.942 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.937+0000 7fa6180b0700 1 -- start start 2023-11-24T06:07:36.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa6180b0700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6100b5eb0 0x7fa610159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:36.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa6180b0700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa6100cc8e0 0x7fa610157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:36.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa6180b0700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa61015f380 0x7fa61009db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:36.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa6180b0700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6100b2840 con 0x7fa61015f380 2023-11-24T06:07:36.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa6180b0700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6100b0940 con 0x7fa6100b5eb0 2023-11-24T06:07:36.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa6180b0700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa610157eb0 con 0x7fa6100cc8e0 2023-11-24T06:07:36.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa615e4c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6100b5eb0 0x7fa610159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:36.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa61564b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa6100cc8e0 0x7fa610157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:36.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa615e4c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6100b5eb0 0x7fa610159ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52212/0 (socket says 172.21.15.94:52212) 2023-11-24T06:07:36.945 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa615e4c700 1 -- 172.21.15.94:0/1461922101 learned_addr learned my addr 172.21.15.94:0/1461922101 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-11-24T06:07:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa61664d700 1 --2- 172.21.15.94:0/1461922101 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa61015f380 0x7fa61009db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa61564b700 1 -- 172.21.15.94:0/1461922101 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6100b5eb0 msgr2=0x7fa610159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa61564b700 1 --2- 172.21.15.94:0/1461922101 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6100b5eb0 0x7fa610159ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa61564b700 1 -- 172.21.15.94:0/1461922101 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa61015f380 msgr2=0x7fa61009db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa61564b700 1 --2- 172.21.15.94:0/1461922101 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa61015f380 0x7fa61009db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa61564b700 1 -- 172.21.15.94:0/1461922101 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa600009550 con 0x7fa6100cc8e0 2023-11-24T06:07:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa615e4c700 1 --2- 172.21.15.94:0/1461922101 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6100b5eb0 0x7fa610159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:07:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa61564b700 1 --2- 172.21.15.94:0/1461922101 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa6100cc8e0 0x7fa610157970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa610072600 tx=0x7fa600004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:36.946 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa6067fc700 1 -- 172.21.15.94:0/1461922101 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa600004380 con 0x7fa6100cc8e0 2023-11-24T06:07:36.947 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa6067fc700 1 -- 172.21.15.94:0/1461922101 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60001f070 con 0x7fa6100cc8e0 2023-11-24T06:07:36.947 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa6067fc700 1 -- 172.21.15.94:0/1461922101 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6000036a0 con 0x7fa6100cc8e0 2023-11-24T06:07:36.947 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.938+0000 7fa6180b0700 1 -- 172.21.15.94:0/1461922101 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61009e0d0 con 0x7fa6100cc8e0 2023-11-24T06:07:36.947 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.939+0000 7fa6180b0700 1 -- 172.21.15.94:0/1461922101 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6100b2180 con 0x7fa6100cc8e0 2023-11-24T06:07:36.947 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.939+0000 7fa6067fc700 1 -- 172.21.15.94:0/1461922101 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa600021030 con 0x7fa6100cc8e0 2023-11-24T06:07:36.947 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.939+0000 7fa6067fc700 1 -- 172.21.15.94:0/1461922101 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fa60001abe0 con 0x7fa6100cc8e0 2023-11-24T06:07:36.947 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.940+0000 7fa6180b0700 1 -- 172.21.15.94:0/1461922101 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa610075480 con 0x7fa6100cc8e0 2023-11-24T06:07:36.948 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:36.942+0000 7fa6067fc700 1 -- 172.21.15.94:0/1461922101 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fa600015020 con 0x7fa6100cc8e0 2023-11-24T06:07:37.009 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:37.005+0000 7ff0604e1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-24T06:07:37.025 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:37.022+0000 7f17cbd25200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-24T06:07:37.035 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.032+0000 7fa6180b0700 1 -- 172.21.15.94:0/1461922101 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6938ed71-88b6-47e4-9c1d-fbc2856d74f9", "id": 2} v 0) v1 -- 0x7fa61015a0a0 con 0x7fa6100cc8e0 2023-11-24T06:07:37.040 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.037+0000 7fa6067fc700 1 -- 172.21.15.94:0/1461922101 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6938ed71-88b6-47e4-9c1d-fbc2856d74f9", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa60001ae00 con 0x7fa6100cc8e0 2023-11-24T06:07:37.041 INFO:teuthology.orchestra.run.smithi094.stdout:2 2023-11-24T06:07:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.038+0000 7fa6180b0700 1 -- 172.21.15.94:0/1461922101 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa6100cc8e0 msgr2=0x7fa610157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.038+0000 7fa6180b0700 1 --2- 172.21.15.94:0/1461922101 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa6100cc8e0 0x7fa610157970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa610072600 tx=0x7fa600004750 comp rx=0 tx=0).stop 2023-11-24T06:07:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.038+0000 7fa6180b0700 1 -- 172.21.15.94:0/1461922101 shutdown_connections 2023-11-24T06:07:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.038+0000 7fa6180b0700 1 --2- 172.21.15.94:0/1461922101 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa6100b5eb0 0x7fa610159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.038+0000 7fa6180b0700 1 --2- 172.21.15.94:0/1461922101 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa61015f380 0x7fa61009db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.038+0000 7fa6180b0700 1 --2- 172.21.15.94:0/1461922101 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa6100cc8e0 0x7fa610157970 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.038+0000 7fa6180b0700 1 -- 172.21.15.94:0/1461922101 >> 172.21.15.94:0/1461922101 conn(0x7fa6100b5070 msgr2=0x7fa6100cf810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.038+0000 7fa6180b0700 1 -- 172.21.15.94:0/1461922101 shutdown_connections 2023-11-24T06:07:37.042 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.038+0000 7fa6180b0700 1 -- 172.21.15.94:0/1461922101 wait complete. 2023-11-24T06:07:37.053 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 07a912f1-95b0-4426-85d6-61f77bfa99bf 3 2023-11-24T06:07:37.218 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.215+0000 7f02452c5700 1 Processor -- start 2023-11-24T06:07:37.219 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.215+0000 7f02452c5700 1 -- start start 2023-11-24T06:07:37.222 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.216+0000 7f02452c5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f02400b4090 0x7f024015e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.222 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.216+0000 7f02452c5700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f0240072a00 0x7f024006db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.222 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.216+0000 7f02452c5700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02400d0b80 con 0x7f024006e0a0 2023-11-24T06:07:37.222 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.216+0000 7f02452c5700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f024015ed60 con 0x7f0240072a00 2023-11-24T06:07:37.222 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.216+0000 7f02452c5700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024015f040 con 0x7f02400b4090 2023-11-24T06:07:37.223 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.216+0000 7f023effd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f02400b4090 0x7f024015e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.223 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.216+0000 7f023f7fe700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f024006e0a0 0x7f02400d1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.94:33382/0 (socket says 172.21.15.94:33382) 2023-11-24T06:07:37.223 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.216+0000 7f023effd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f02400b4090 0x7f024015e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:50834/0 (socket says 172.21.15.94:50834) 2023-11-24T06:07:37.223 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.216+0000 7f023f7fe700 1 -- 172.21.15.94:0/4079093090 learned_addr learned my addr 172.21.15.94:0/4079093090 (peer_addr_for_me v1:172.21.15.94:0/0) 2023-11-24T06:07:37.223 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.216+0000 7f023e7fc700 1 --2- 172.21.15.94:0/4079093090 >> v2:172.21.15.91:3300/0 conn(0x7f0240072a00 0x7f024006db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.224 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.216+0000 7f023effd700 1 -- 172.21.15.94:0/4079093090 >> v2:172.21.15.91:3300/0 conn(0x7f0240072a00 msgr2=0x7f024006db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:37.224 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.216+0000 7f023effd700 1 --2- 172.21.15.94:0/4079093090 >> v2:172.21.15.91:3300/0 conn(0x7f0240072a00 0x7f024006db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.224 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.216+0000 7f023effd700 1 -- 172.21.15.94:0/4079093090 >> v1:172.21.15.94:6789/0 conn(0x7f024006e0a0 legacy=0x7f02400d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.224 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.216+0000 7f023effd700 1 -- 172.21.15.94:0/4079093090 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024015f320 con 0x7f02400b4090 2023-11-24T06:07:37.224 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.216+0000 7f023dffb700 1 -- 172.21.15.94:0/4079093090 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077878550 0 0) 0x7f02400d0b80 con 0x7f024006e0a0 2023-11-24T06:07:37.224 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.216+0000 7f023e7fc700 1 --2- 172.21.15.94:0/4079093090 >> v2:172.21.15.91:3300/0 conn(0x7f0240072a00 0x7f024006db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:07:37.225 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.217+0000 7f023effd700 1 --2- 172.21.15.94:0/4079093090 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f02400b4090 0x7f024015e510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0230009840 tx=0x7f0230009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48e5dcddb27881e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:37.225 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.217+0000 7f023dffb700 1 -- 172.21.15.94:0/4079093090 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0230011830 con 0x7f02400b4090 2023-11-24T06:07:37.225 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.217+0000 7f023dffb700 1 -- 172.21.15.94:0/4079093090 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0230011990 con 0x7f02400b4090 2023-11-24T06:07:37.225 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.217+0000 7f023dffb700 1 -- 172.21.15.94:0/4079093090 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0230011d20 con 0x7f02400b4090 2023-11-24T06:07:37.225 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.217+0000 7f02452c5700 1 -- 172.21.15.94:0/4079093090 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f02400b4090 msgr2=0x7f024015e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.217+0000 7f02452c5700 1 --2- 172.21.15.94:0/4079093090 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f02400b4090 0x7f024015e510 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0230009840 tx=0x7f0230009b50 comp rx=0 tx=0).stop 2023-11-24T06:07:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.217+0000 7f02452c5700 1 -- 172.21.15.94:0/4079093090 shutdown_connections 2023-11-24T06:07:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.217+0000 7f02452c5700 1 --2- 172.21.15.94:0/4079093090 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f02400b4090 0x7f024015e510 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.217+0000 7f02452c5700 1 --2- 172.21.15.94:0/4079093090 >> v2:172.21.15.91:3300/0 conn(0x7f0240072a00 0x7f024006db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.217+0000 7f02452c5700 1 -- 172.21.15.94:0/4079093090 >> 172.21.15.94:0/4079093090 conn(0x7f02400b2740 msgr2=0x7f02400b2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:37.226 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.217+0000 7f02452c5700 1 -- 172.21.15.94:0/4079093090 shutdown_connections 2023-11-24T06:07:37.227 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.217+0000 7f02452c5700 1 -- 172.21.15.94:0/4079093090 wait complete. 2023-11-24T06:07:37.227 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.219+0000 7f02452c5700 1 Processor -- start 2023-11-24T06:07:37.227 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.219+0000 7f02452c5700 1 -- start start 2023-11-24T06:07:37.230 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.219+0000 7f02452c5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f024006e0a0 0x7f0240158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.231 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.219+0000 7f02452c5700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0240072a00 0x7f024015b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.231 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.219+0000 7f02452c5700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f02400b4090 0x7f0240159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.231 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.219+0000 7f02452c5700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0240156880 con 0x7f02400b4090 2023-11-24T06:07:37.231 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.219+0000 7f02452c5700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0240155c70 con 0x7f0240072a00 2023-11-24T06:07:37.231 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.219+0000 7f02452c5700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0240155f50 con 0x7f024006e0a0 2023-11-24T06:07:37.231 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f023effd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f024006e0a0 0x7f0240158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.232 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f023e7fc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0240072a00 0x7f024015b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.232 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f023effd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f024006e0a0 0x7f0240158c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:50840/0 (socket says 172.21.15.94:50840) 2023-11-24T06:07:37.232 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f023effd700 1 -- 172.21.15.94:0/1048275022 learned_addr learned my addr 172.21.15.94:0/1048275022 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-11-24T06:07:37.232 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f023f7fe700 1 --2- 172.21.15.94:0/1048275022 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f02400b4090 0x7f0240159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.232 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f023e7fc700 1 -- 172.21.15.94:0/1048275022 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f024006e0a0 msgr2=0x7f0240158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.232 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f023e7fc700 1 --2- 172.21.15.94:0/1048275022 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f024006e0a0 0x7f0240158c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.232 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f023e7fc700 1 -- 172.21.15.94:0/1048275022 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f02400b4090 msgr2=0x7f0240159420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.232 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f023e7fc700 1 --2- 172.21.15.94:0/1048275022 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f02400b4090 0x7f0240159420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.232 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f023e7fc700 1 -- 172.21.15.94:0/1048275022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02300094a0 con 0x7f0240072a00 2023-11-24T06:07:37.232 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f023e7fc700 1 --2- 172.21.15.94:0/1048275022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0240072a00 0x7f024015b410 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f023400e810 tx=0x7f023400eb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:37.232 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f02277fe700 1 -- 172.21.15.94:0/1048275022 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023400c070 con 0x7f0240072a00 2023-11-24T06:07:37.233 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f02277fe700 1 -- 172.21.15.94:0/1048275022 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0234004230 con 0x7f0240072a00 2023-11-24T06:07:37.233 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f02277fe700 1 -- 172.21.15.94:0/1048275022 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02340036a0 con 0x7f0240072a00 2023-11-24T06:07:37.233 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f02452c5700 1 -- 172.21.15.94:0/1048275022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0240156230 con 0x7f0240072a00 2023-11-24T06:07:37.233 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f02452c5700 1 -- 172.21.15.94:0/1048275022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f024015b950 con 0x7f0240072a00 2023-11-24T06:07:37.233 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f02277fe700 1 -- 172.21.15.94:0/1048275022 <== mon.1 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0234003b00 con 0x7f0240072a00 2023-11-24T06:07:37.233 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.220+0000 7f02277fe700 1 -- 172.21.15.94:0/1048275022 <== mon.1 v2:172.21.15.94:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f023401c400 con 0x7f0240072a00 2023-11-24T06:07:37.233 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.221+0000 7f02452c5700 1 -- 172.21.15.94:0/1048275022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0240156a30 con 0x7f0240072a00 2023-11-24T06:07:37.233 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.223+0000 7f02277fe700 1 -- 172.21.15.94:0/1048275022 <== mon.1 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f0240156a30 con 0x7f0240072a00 2023-11-24T06:07:37.312 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.308+0000 7f02452c5700 1 -- 172.21.15.94:0/1048275022 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd new", "uuid": "07a912f1-95b0-4426-85d6-61f77bfa99bf", "id": 3} v 0) v1 -- 0x7f0240075480 con 0x7f0240072a00 2023-11-24T06:07:37.315 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.311+0000 7f02277fe700 1 -- 172.21.15.94:0/1048275022 <== mon.1 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "07a912f1-95b0-4426-85d6-61f77bfa99bf", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f023400f040 con 0x7f0240072a00 2023-11-24T06:07:37.315 INFO:teuthology.orchestra.run.smithi094.stdout:3 2023-11-24T06:07:37.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.312+0000 7f02452c5700 1 -- 172.21.15.94:0/1048275022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0240072a00 msgr2=0x7f024015b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.312+0000 7f02452c5700 1 --2- 172.21.15.94:0/1048275022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0240072a00 0x7f024015b410 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f023400e810 tx=0x7f023400eb20 comp rx=0 tx=0).stop 2023-11-24T06:07:37.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.312+0000 7f02452c5700 1 -- 172.21.15.94:0/1048275022 shutdown_connections 2023-11-24T06:07:37.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.312+0000 7f02452c5700 1 --2- 172.21.15.94:0/1048275022 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f02400b4090 0x7f0240159420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.312+0000 7f02452c5700 1 --2- 172.21.15.94:0/1048275022 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f024006e0a0 0x7f0240158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.312+0000 7f02452c5700 1 --2- 172.21.15.94:0/1048275022 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0240072a00 0x7f024015b410 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.312+0000 7f02452c5700 1 -- 172.21.15.94:0/1048275022 >> 172.21.15.94:0/1048275022 conn(0x7f02400b2740 msgr2=0x7f024006afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:37.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.312+0000 7f02452c5700 1 -- 172.21.15.94:0/1048275022 shutdown_connections 2023-11-24T06:07:37.316 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.312+0000 7f02452c5700 1 -- 172.21.15.94:0/1048275022 wait complete. 2023-11-24T06:07:37.328 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 5f1c8835-4980-4ced-8b44-fb6bebe8a777 4 2023-11-24T06:07:37.439 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:37.435+0000 7ff0604e1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-24T06:07:37.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.446+0000 7f441aef1700 1 Processor -- start 2023-11-24T06:07:37.449 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.446+0000 7f441aef1700 1 -- start start 2023-11-24T06:07:37.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f441aef1700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f441406b620 0x7f44140b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f441aef1700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f44140b5ab0 0x7f44140b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f441aef1700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f44141478c0 0x7f4414161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f441aef1700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4414071f50 con 0x7f44140b5ab0 2023-11-24T06:07:37.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f441aef1700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4414072230 con 0x7f441406b620 2023-11-24T06:07:37.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f441aef1700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4414072510 con 0x7f44141478c0 2023-11-24T06:07:37.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f4418c8d700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f441406b620 0x7f44140b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.452 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f441948e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f44141478c0 0x7f4414161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f4418c8d700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f441406b620 0x7f44140b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52230/0 (socket says 172.21.15.94:52230) 2023-11-24T06:07:37.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f441948e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f44141478c0 0x7f4414161ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:50850/0 (socket says 172.21.15.94:50850) 2023-11-24T06:07:37.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f4418c8d700 1 -- 172.21.15.94:0/3909065041 learned_addr learned my addr 172.21.15.94:0/3909065041 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-11-24T06:07:37.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f4413fff700 1 --2- 172.21.15.94:0/3909065041 >> v2:172.21.15.91:3300/0 conn(0x7f44140b5ab0 0x7f44140b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f4418c8d700 1 -- 172.21.15.94:0/3909065041 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f44141478c0 msgr2=0x7f4414161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:37.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f4418c8d700 1 --2- 172.21.15.94:0/3909065041 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f44141478c0 0x7f4414161ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f4418c8d700 1 -- 172.21.15.94:0/3909065041 >> v2:172.21.15.91:3300/0 conn(0x7f44140b5ab0 msgr2=0x7f44140b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:37.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f4418c8d700 1 --2- 172.21.15.94:0/3909065041 >> v2:172.21.15.91:3300/0 conn(0x7f44140b5ab0 0x7f44140b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f4418c8d700 1 -- 172.21.15.94:0/3909065041 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44140727f0 con 0x7f441406b620 2023-11-24T06:07:37.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f441948e700 1 --2- 172.21.15.94:0/3909065041 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f44141478c0 0x7f4414161ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:07:37.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f4418c8d700 1 --2- 172.21.15.94:0/3909065041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f441406b620 0x7f44140b3a40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f441406d2e0 tx=0x7f4408009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f88d2d599707b975 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:37.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f44137fe700 1 -- 172.21.15.94:0/3909065041 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4408011830 con 0x7f441406b620 2023-11-24T06:07:37.453 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f44137fe700 1 -- 172.21.15.94:0/3909065041 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4408011990 con 0x7f441406b620 2023-11-24T06:07:37.454 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.447+0000 7f44137fe700 1 -- 172.21.15.94:0/3909065041 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4408011d20 con 0x7f441406b620 2023-11-24T06:07:37.454 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.448+0000 7f441aef1700 1 -- 172.21.15.94:0/3909065041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f441406b620 msgr2=0x7f44140b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.454 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.448+0000 7f441aef1700 1 --2- 172.21.15.94:0/3909065041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f441406b620 0x7f44140b3a40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f441406d2e0 tx=0x7f4408009bd0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.454 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.448+0000 7f441aef1700 1 -- 172.21.15.94:0/3909065041 shutdown_connections 2023-11-24T06:07:37.454 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.448+0000 7f441aef1700 1 --2- 172.21.15.94:0/3909065041 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f44141478c0 0x7f4414161ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.454 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.448+0000 7f441aef1700 1 --2- 172.21.15.94:0/3909065041 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f441406b620 0x7f44140b3a40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.454 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.448+0000 7f441aef1700 1 --2- 172.21.15.94:0/3909065041 >> v2:172.21.15.91:3300/0 conn(0x7f44140b5ab0 0x7f44140b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.454 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.448+0000 7f441aef1700 1 -- 172.21.15.94:0/3909065041 >> 172.21.15.94:0/3909065041 conn(0x7f44140b2740 msgr2=0x7f44140b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:37.454 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.448+0000 7f441aef1700 1 -- 172.21.15.94:0/3909065041 shutdown_connections 2023-11-24T06:07:37.454 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.448+0000 7f441aef1700 1 -- 172.21.15.94:0/3909065041 wait complete. 2023-11-24T06:07:37.457 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.449+0000 7f441aef1700 1 Processor -- start 2023-11-24T06:07:37.457 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.449+0000 7f441aef1700 1 -- start start 2023-11-24T06:07:37.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.450+0000 7f441aef1700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f441406b620 0x7f44141433d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.450+0000 7f441aef1700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44140b5ab0 0x7f44141e9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.450+0000 7f441aef1700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f44141478c0 0x7f44140c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.450+0000 7f441aef1700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4414143910 con 0x7f441406b620 2023-11-24T06:07:37.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.450+0000 7f441aef1700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44141e90a0 con 0x7f44140b5ab0 2023-11-24T06:07:37.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.450+0000 7f441aef1700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44141e7d20 con 0x7f44141478c0 2023-11-24T06:07:37.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.450+0000 7f441948e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f44141478c0 0x7f44140c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.450+0000 7f4413fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44140b5ab0 0x7f44141e9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.458 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.450+0000 7f4418c8d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f441406b620 0x7f44141433d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.459 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.450+0000 7f441948e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f44141478c0 0x7f44140c5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:50856/0 (socket says 172.21.15.94:50856) 2023-11-24T06:07:37.459 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.450+0000 7f4413fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44140b5ab0 0x7f44141e9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52238/0 (socket says 172.21.15.94:52238) 2023-11-24T06:07:37.459 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.450+0000 7f441948e700 1 -- 172.21.15.94:0/1115252279 learned_addr learned my addr 172.21.15.94:0/1115252279 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-11-24T06:07:37.459 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.451+0000 7f4413fff700 1 -- 172.21.15.94:0/1115252279 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f44141478c0 msgr2=0x7f44140c5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.459 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.451+0000 7f4413fff700 1 --2- 172.21.15.94:0/1115252279 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f44141478c0 0x7f44140c5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.459 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.451+0000 7f4413fff700 1 -- 172.21.15.94:0/1115252279 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f441406b620 msgr2=0x7f44141433d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.459 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.451+0000 7f4413fff700 1 --2- 172.21.15.94:0/1115252279 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f441406b620 0x7f44141433d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.459 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.451+0000 7f4413fff700 1 -- 172.21.15.94:0/1115252279 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440c009a70 con 0x7f44140b5ab0 2023-11-24T06:07:37.459 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.451+0000 7f441948e700 1 --2- 172.21.15.94:0/1115252279 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f44141478c0 0x7f44140c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:07:37.459 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.451+0000 7f4413fff700 1 --2- 172.21.15.94:0/1115252279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44140b5ab0 0x7f44141e9770 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f440000e840 tx=0x7f440000eb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:37.459 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.451+0000 7f44117fa700 1 -- 172.21.15.94:0/1115252279 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4400013680 con 0x7f44140b5ab0 2023-11-24T06:07:37.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.451+0000 7f44117fa700 1 -- 172.21.15.94:0/1115252279 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44000137e0 con 0x7f44140b5ab0 2023-11-24T06:07:37.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.451+0000 7f441aef1700 1 -- 172.21.15.94:0/1115252279 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4408009550 con 0x7f44140b5ab0 2023-11-24T06:07:37.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.451+0000 7f44117fa700 1 -- 172.21.15.94:0/1115252279 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4400013b50 con 0x7f44140b5ab0 2023-11-24T06:07:37.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.451+0000 7f441aef1700 1 -- 172.21.15.94:0/1115252279 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44141e82e0 con 0x7f44140b5ab0 2023-11-24T06:07:37.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.451+0000 7f44117fa700 1 -- 172.21.15.94:0/1115252279 <== mon.1 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4400005a20 con 0x7f44140b5ab0 2023-11-24T06:07:37.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.451+0000 7f44117fa700 1 -- 172.21.15.94:0/1115252279 <== mon.1 v2:172.21.15.94:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f4400006750 con 0x7f44140b5ab0 2023-11-24T06:07:37.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.452+0000 7f441aef1700 1 -- 172.21.15.94:0/1115252279 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4414075480 con 0x7f44140b5ab0 2023-11-24T06:07:37.460 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.454+0000 7f44117fa700 1 -- 172.21.15.94:0/1115252279 <== mon.1 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f4400021060 con 0x7f44140b5ab0 2023-11-24T06:07:37.469 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:37.466+0000 7f17cbd25200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-24T06:07:37.543 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.540+0000 7f441aef1700 1 -- 172.21.15.94:0/1115252279 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5f1c8835-4980-4ced-8b44-fb6bebe8a777", "id": 4} v 0) v1 -- 0x7f441406bd40 con 0x7f44140b5ab0 2023-11-24T06:07:37.548 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.545+0000 7f44117fa700 1 -- 172.21.15.94:0/1115252279 <== mon.1 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5f1c8835-4980-4ced-8b44-fb6bebe8a777", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4400017070 con 0x7f44140b5ab0 2023-11-24T06:07:37.549 INFO:teuthology.orchestra.run.smithi094.stdout:4 2023-11-24T06:07:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.546+0000 7f441aef1700 1 -- 172.21.15.94:0/1115252279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44140b5ab0 msgr2=0x7f44141e9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.546+0000 7f441aef1700 1 --2- 172.21.15.94:0/1115252279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44140b5ab0 0x7f44141e9770 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f440000e840 tx=0x7f440000eb50 comp rx=0 tx=0).stop 2023-11-24T06:07:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.546+0000 7f441aef1700 1 -- 172.21.15.94:0/1115252279 shutdown_connections 2023-11-24T06:07:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.546+0000 7f441aef1700 1 --2- 172.21.15.94:0/1115252279 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f44141478c0 0x7f44140c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.546+0000 7f441aef1700 1 --2- 172.21.15.94:0/1115252279 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f441406b620 0x7f44141433d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.546+0000 7f441aef1700 1 --2- 172.21.15.94:0/1115252279 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f44140b5ab0 0x7f44141e9770 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.546+0000 7f441aef1700 1 -- 172.21.15.94:0/1115252279 >> 172.21.15.94:0/1115252279 conn(0x7f44140b2740 msgr2=0x7f44140b1470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.546+0000 7f441aef1700 1 -- 172.21.15.94:0/1115252279 shutdown_connections 2023-11-24T06:07:37.550 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.546+0000 7f441aef1700 1 -- 172.21.15.94:0/1115252279 wait complete. 2023-11-24T06:07:37.561 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new de892401-7f4e-423f-9447-4a750970534f 5 2023-11-24T06:07:37.680 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.677+0000 7f87a3c49700 1 Processor -- start 2023-11-24T06:07:37.681 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.677+0000 7f87a3c49700 1 -- start start 2023-11-24T06:07:37.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.678+0000 7f87a3c49700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f879c15bde0 0x7f879c15c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.678+0000 7f87a3c49700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f879c0d0a90 con 0x7f879c0cf2b0 2023-11-24T06:07:37.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.678+0000 7f87a3c49700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f879c06ee10 con 0x7f879c0ce5c0 2023-11-24T06:07:37.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.678+0000 7f87a3c49700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f879c06f0f0 con 0x7f879c15bde0 2023-11-24T06:07:37.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.678+0000 7f87a19e5700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f879c15bde0 0x7f879c15c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.678+0000 7f87a19e5700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f879c15bde0 0x7f879c15c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52242/0 (socket says 172.21.15.94:52242) 2023-11-24T06:07:37.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.678+0000 7f87a19e5700 1 -- 172.21.15.94:0/2912429188 learned_addr learned my addr 172.21.15.94:0/2912429188 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-11-24T06:07:37.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.679+0000 7f87a19e5700 1 -- 172.21.15.94:0/2912429188 >> v1:172.21.15.94:6789/0 conn(0x7f879c0ce5c0 legacy=0x7f879c06fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.679+0000 7f87a19e5700 1 -- 172.21.15.94:0/2912429188 >> v1:172.21.15.91:6789/0 conn(0x7f879c0cf2b0 legacy=0x7f879c0d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.682 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.679+0000 7f87a19e5700 1 -- 172.21.15.94:0/2912429188 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879c06f3d0 con 0x7f879c15bde0 2023-11-24T06:07:37.683 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.679+0000 7f87a19e5700 1 --2- 172.21.15.94:0/2912429188 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f879c15bde0 0x7f879c15c1c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f87980098f0 tx=0x7f8798009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=222f27d203029f11 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:37.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.679+0000 7f87a09e3700 1 -- 172.21.15.94:0/2912429188 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879800b8a0 con 0x7f879c15bde0 2023-11-24T06:07:37.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.679+0000 7f87a09e3700 1 -- 172.21.15.94:0/2912429188 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879800ba00 con 0x7f879c15bde0 2023-11-24T06:07:37.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.679+0000 7f87a09e3700 1 -- 172.21.15.94:0/2912429188 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879800bd90 con 0x7f879c15bde0 2023-11-24T06:07:37.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.680+0000 7f87a3c49700 1 -- 172.21.15.94:0/2912429188 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f879c15bde0 msgr2=0x7f879c15c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.680+0000 7f87a3c49700 1 --2- 172.21.15.94:0/2912429188 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f879c15bde0 0x7f879c15c1c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f87980098f0 tx=0x7f8798009c00 comp rx=0 tx=0).stop 2023-11-24T06:07:37.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.680+0000 7f87a3c49700 1 -- 172.21.15.94:0/2912429188 shutdown_connections 2023-11-24T06:07:37.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.680+0000 7f87a3c49700 1 --2- 172.21.15.94:0/2912429188 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f879c15bde0 0x7f879c15c1c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.680+0000 7f87a3c49700 1 -- 172.21.15.94:0/2912429188 >> 172.21.15.94:0/2912429188 conn(0x7f879c0b3720 msgr2=0x7f879c0b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:37.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.680+0000 7f87a3c49700 1 -- 172.21.15.94:0/2912429188 shutdown_connections 2023-11-24T06:07:37.684 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.680+0000 7f87a3c49700 1 -- 172.21.15.94:0/2912429188 wait complete. 2023-11-24T06:07:37.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.681+0000 7f87a3c49700 1 Processor -- start 2023-11-24T06:07:37.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.681+0000 7f87a3c49700 1 -- start start 2023-11-24T06:07:37.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a3c49700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f879c0ce5c0 0x7f879c0cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a3c49700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f879c0cf2b0 0x7f879c0c78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a3c49700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f879c15bde0 0x7f879c14bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a3c49700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879c0cb510 con 0x7f879c15bde0 2023-11-24T06:07:37.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a3c49700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879c0ca890 con 0x7f879c0ce5c0 2023-11-24T06:07:37.687 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a3c49700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f879c0cab70 con 0x7f879c0cf2b0 2023-11-24T06:07:37.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a11e4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f879c0cf2b0 0x7f879c0c78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a11e4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f879c0cf2b0 0x7f879c0c78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:50866/0 (socket says 172.21.15.94:50866) 2023-11-24T06:07:37.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a11e4700 1 -- 172.21.15.94:0/934857761 learned_addr learned my addr 172.21.15.94:0/934857761 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-11-24T06:07:37.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a19e5700 1 --2- 172.21.15.94:0/934857761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f879c0ce5c0 0x7f879c0cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a21e6700 1 --2- 172.21.15.94:0/934857761 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f879c15bde0 0x7f879c14bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a11e4700 1 -- 172.21.15.94:0/934857761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f879c0ce5c0 msgr2=0x7f879c0cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a11e4700 1 --2- 172.21.15.94:0/934857761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f879c0ce5c0 0x7f879c0cb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a11e4700 1 -- 172.21.15.94:0/934857761 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f879c15bde0 msgr2=0x7f879c14bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a11e4700 1 --2- 172.21.15.94:0/934857761 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f879c15bde0 0x7f879c14bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a11e4700 1 -- 172.21.15.94:0/934857761 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8798009550 con 0x7f879c0cf2b0 2023-11-24T06:07:37.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a19e5700 1 --2- 172.21.15.94:0/934857761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f879c0ce5c0 0x7f879c0cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:07:37.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a11e4700 1 --2- 172.21.15.94:0/934857761 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f879c0cf2b0 0x7f879c0c78c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f878c00c4b0 tx=0x7f878c00c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:37.688 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87927fc700 1 -- 172.21.15.94:0/934857761 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c0052d0 con 0x7f879c0cf2b0 2023-11-24T06:07:37.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87927fc700 1 -- 172.21.15.94:0/934857761 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878c005430 con 0x7f879c0cf2b0 2023-11-24T06:07:37.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87927fc700 1 -- 172.21.15.94:0/934857761 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878c0038c0 con 0x7f879c0cf2b0 2023-11-24T06:07:37.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a3c49700 1 -- 172.21.15.94:0/934857761 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f879c0cae50 con 0x7f879c0cf2b0 2023-11-24T06:07:37.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.682+0000 7f87a3c49700 1 -- 172.21.15.94:0/934857761 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f879c14c1c0 con 0x7f879c0cf2b0 2023-11-24T06:07:37.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.683+0000 7f87927fc700 1 -- 172.21.15.94:0/934857761 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f878c003d20 con 0x7f879c0cf2b0 2023-11-24T06:07:37.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.683+0000 7f87927fc700 1 -- 172.21.15.94:0/934857761 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f878c011d10 con 0x7f879c0cf2b0 2023-11-24T06:07:37.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.683+0000 7f87a3c49700 1 -- 172.21.15.94:0/934857761 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f879c075480 con 0x7f879c0cf2b0 2023-11-24T06:07:37.689 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.685+0000 7f87927fc700 1 -- 172.21.15.94:0/934857761 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f878c017070 con 0x7f879c0cf2b0 2023-11-24T06:07:37.774 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.771+0000 7f87a3c49700 1 -- 172.21.15.94:0/934857761 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd new", "uuid": "de892401-7f4e-423f-9447-4a750970534f", "id": 5} v 0) v1 -- 0x7f879c0cb650 con 0x7f879c0cf2b0 2023-11-24T06:07:37.779 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.776+0000 7f87927fc700 1 -- 172.21.15.94:0/934857761 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "de892401-7f4e-423f-9447-4a750970534f", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f879c0cb650 con 0x7f879c0cf2b0 2023-11-24T06:07:37.779 INFO:teuthology.orchestra.run.smithi094.stdout:5 2023-11-24T06:07:37.781 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.777+0000 7f87a3c49700 1 -- 172.21.15.94:0/934857761 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f879c0cf2b0 msgr2=0x7f879c0c78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.781 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.777+0000 7f87a3c49700 1 --2- 172.21.15.94:0/934857761 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f879c0cf2b0 0x7f879c0c78c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f878c00c4b0 tx=0x7f878c00c7c0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.781 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.777+0000 7f87a3c49700 1 -- 172.21.15.94:0/934857761 shutdown_connections 2023-11-24T06:07:37.781 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.777+0000 7f87a3c49700 1 --2- 172.21.15.94:0/934857761 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f879c0ce5c0 0x7f879c0cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.781 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.777+0000 7f87a3c49700 1 --2- 172.21.15.94:0/934857761 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f879c15bde0 0x7f879c14bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.781 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.777+0000 7f87a3c49700 1 --2- 172.21.15.94:0/934857761 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f879c0cf2b0 0x7f879c0c78c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.781 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.777+0000 7f87a3c49700 1 -- 172.21.15.94:0/934857761 >> 172.21.15.94:0/934857761 conn(0x7f879c0b3720 msgr2=0x7f879c0af910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:37.781 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.777+0000 7f87a3c49700 1 -- 172.21.15.94:0/934857761 shutdown_connections 2023-11-24T06:07:37.781 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.777+0000 7f87a3c49700 1 -- 172.21.15.94:0/934857761 wait complete. 2023-11-24T06:07:37.792 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71 6 2023-11-24T06:07:37.946 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.943+0000 7fc709d70700 1 Processor -- start 2023-11-24T06:07:37.946 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.943+0000 7fc709d70700 1 -- start start 2023-11-24T06:07:37.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc709d70700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc7040d0870 0x7fc7040d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc709d70700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fc7040d0c50 0x7fc7040d49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc709d70700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7040b5370 con 0x7fc7040cf860 2023-11-24T06:07:37.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc709d70700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7040d2f80 con 0x7fc7040d0c50 2023-11-24T06:07:37.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc709d70700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7040ceae0 con 0x7fc7040d0870 2023-11-24T06:07:37.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc7037fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc7040d0870 0x7fc7040d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc7037fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc7040d0870 0x7fc7040d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52254/0 (socket says 172.21.15.94:52254) 2023-11-24T06:07:37.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc7037fe700 1 -- 172.21.15.94:0/3549913124 learned_addr learned my addr 172.21.15.94:0/3549913124 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-11-24T06:07:37.949 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc6fbfff700 1 --2- 172.21.15.94:0/3549913124 >> v2:172.21.15.91:3300/0 conn(0x7fc7040d0c50 0x7fc7040d49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.950 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc7037fe700 1 -- 172.21.15.94:0/3549913124 >> v2:172.21.15.91:3300/0 conn(0x7fc7040d0c50 msgr2=0x7fc7040d49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:37.950 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc7037fe700 1 --2- 172.21.15.94:0/3549913124 >> v2:172.21.15.91:3300/0 conn(0x7fc7040d0c50 0x7fc7040d49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.950 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc7037fe700 1 -- 172.21.15.94:0/3549913124 >> v1:172.21.15.94:6789/0 conn(0x7fc7040cf860 legacy=0x7fc7040b6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.950 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc7037fe700 1 -- 172.21.15.94:0/3549913124 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7040d2ca0 con 0x7fc7040d0870 2023-11-24T06:07:37.950 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc7037fe700 1 --2- 172.21.15.94:0/3549913124 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7040d0870 0x7fc7040d26c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc70408c560 tx=0x7fc6f4009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a8fee1fb4774d47 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:37.950 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc702ffd700 1 -- 172.21.15.94:0/3549913124 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4011830 con 0x7fc7040d0870 2023-11-24T06:07:37.950 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc702ffd700 1 -- 172.21.15.94:0/3549913124 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f4011990 con 0x7fc7040d0870 2023-11-24T06:07:37.950 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.944+0000 7fc702ffd700 1 -- 172.21.15.94:0/3549913124 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4011d20 con 0x7fc7040d0870 2023-11-24T06:07:37.950 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.945+0000 7fc709d70700 1 -- 172.21.15.94:0/3549913124 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7040d0870 msgr2=0x7fc7040d26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.950 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.945+0000 7fc709d70700 1 --2- 172.21.15.94:0/3549913124 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7040d0870 0x7fc7040d26c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc70408c560 tx=0x7fc6f4009bd0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.950 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.945+0000 7fc709d70700 1 -- 172.21.15.94:0/3549913124 shutdown_connections 2023-11-24T06:07:37.950 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.945+0000 7fc709d70700 1 --2- 172.21.15.94:0/3549913124 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7040d0870 0x7fc7040d26c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.951 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.945+0000 7fc709d70700 1 --2- 172.21.15.94:0/3549913124 >> v2:172.21.15.91:3300/0 conn(0x7fc7040d0c50 0x7fc7040d49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.951 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.945+0000 7fc709d70700 1 -- 172.21.15.94:0/3549913124 >> 172.21.15.94:0/3549913124 conn(0x7fc7040d3ed0 msgr2=0x7fc7040d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:37.951 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.945+0000 7fc709d70700 1 -- 172.21.15.94:0/3549913124 shutdown_connections 2023-11-24T06:07:37.951 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.945+0000 7fc709d70700 1 -- 172.21.15.94:0/3549913124 wait complete. 2023-11-24T06:07:37.953 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.947+0000 7fc709d70700 1 Processor -- start 2023-11-24T06:07:37.953 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.947+0000 7fc709d70700 1 -- start start 2023-11-24T06:07:37.953 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.948+0000 7fc709d70700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7040cf860 0x7fc704154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.953 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.948+0000 7fc709d70700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc7040d0870 0x7fc704156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.953 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.948+0000 7fc709d70700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7040d0c50 0x7fc70414b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:37.954 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.948+0000 7fc709d70700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc704157350 con 0x7fc7040cf860 2023-11-24T06:07:37.954 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.948+0000 7fc709d70700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc704156ab0 con 0x7fc7040d0c50 2023-11-24T06:07:37.954 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.948+0000 7fc709d70700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc704156d90 con 0x7fc7040d0870 2023-11-24T06:07:37.954 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.948+0000 7fc703fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7040d0c50 0x7fc70414b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.954 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.948+0000 7fc6fbfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc7040d0870 0x7fc704156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.954 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.948+0000 7fc7037fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7040cf860 0x7fc704154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:37.954 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.948+0000 7fc703fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7040d0c50 0x7fc70414b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52264/0 (socket says 172.21.15.94:52264) 2023-11-24T06:07:37.954 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.948+0000 7fc6fbfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc7040d0870 0x7fc704156280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.94:50878/0 (socket says 172.21.15.94:50878) 2023-11-24T06:07:37.954 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.948+0000 7fc7037fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7040cf860 0x7fc704154220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.94:35498/0 (socket says 172.21.15.94:35498) 2023-11-24T06:07:37.954 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.948+0000 7fc703fff700 1 -- 172.21.15.94:0/1674080250 learned_addr learned my addr 172.21.15.94:0/1674080250 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-11-24T06:07:37.955 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.949+0000 7fc703fff700 1 -- 172.21.15.94:0/1674080250 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc7040d0870 msgr2=0x7fc704156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.955 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.949+0000 7fc703fff700 1 --2- 172.21.15.94:0/1674080250 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc7040d0870 0x7fc704156280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.955 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.949+0000 7fc703fff700 1 -- 172.21.15.94:0/1674080250 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7040cf860 msgr2=0x7fc704154220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:37.955 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.949+0000 7fc703fff700 1 --2- 172.21.15.94:0/1674080250 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7040cf860 0x7fc704154220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:37.955 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.949+0000 7fc703fff700 1 -- 172.21.15.94:0/1674080250 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f4009550 con 0x7fc7040d0c50 2023-11-24T06:07:37.955 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.949+0000 7fc703fff700 1 --2- 172.21.15.94:0/1674080250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7040d0c50 0x7fc70414b500 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00cb50 tx=0x7fc6fc00ce60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:37.955 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.949+0000 7fc700ff9700 1 -- 172.21.15.94:0/1674080250 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc003c00 con 0x7fc7040d0c50 2023-11-24T06:07:37.955 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.949+0000 7fc700ff9700 1 -- 172.21.15.94:0/1674080250 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6fc003d60 con 0x7fc7040d0c50 2023-11-24T06:07:37.955 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.949+0000 7fc709d70700 1 -- 172.21.15.94:0/1674080250 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc704157070 con 0x7fc7040d0c50 2023-11-24T06:07:37.955 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.949+0000 7fc700ff9700 1 -- 172.21.15.94:0/1674080250 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc00aa20 con 0x7fc7040d0c50 2023-11-24T06:07:37.955 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.949+0000 7fc709d70700 1 -- 172.21.15.94:0/1674080250 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7041567c0 con 0x7fc7040d0c50 2023-11-24T06:07:37.955 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.949+0000 7fc700ff9700 1 -- 172.21.15.94:0/1674080250 <== mon.1 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc6fc0050d0 con 0x7fc7040d0c50 2023-11-24T06:07:37.955 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.949+0000 7fc700ff9700 1 -- 172.21.15.94:0/1674080250 <== mon.1 v2:172.21.15.94:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fc6fc012ba0 con 0x7fc7040d0c50 2023-11-24T06:07:37.956 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.950+0000 7fc709d70700 1 -- 172.21.15.94:0/1674080250 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7040b57c0 con 0x7fc7040d0c50 2023-11-24T06:07:37.957 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:37.954+0000 7fc700ff9700 1 -- 172.21.15.94:0/1674080250 <== mon.1 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fc6fc012e10 con 0x7fc7040d0c50 2023-11-24T06:07:38.046 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.043+0000 7fc709d70700 1 -- 172.21.15.94:0/1674080250 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71", "id": 6} v 0) v1 -- 0x7fc704067a20 con 0x7fc7040d0c50 2023-11-24T06:07:38.051 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.048+0000 7fc700ff9700 1 -- 172.21.15.94:0/1674080250 <== mon.1 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc6fc0125b0 con 0x7fc7040d0c50 2023-11-24T06:07:38.052 INFO:teuthology.orchestra.run.smithi094.stdout:6 2023-11-24T06:07:38.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.049+0000 7fc709d70700 1 -- 172.21.15.94:0/1674080250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7040d0c50 msgr2=0x7fc70414b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:38.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.049+0000 7fc709d70700 1 --2- 172.21.15.94:0/1674080250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7040d0c50 0x7fc70414b500 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00cb50 tx=0x7fc6fc00ce60 comp rx=0 tx=0).stop 2023-11-24T06:07:38.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.050+0000 7fc709d70700 1 -- 172.21.15.94:0/1674080250 shutdown_connections 2023-11-24T06:07:38.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.050+0000 7fc709d70700 1 --2- 172.21.15.94:0/1674080250 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc7040d0c50 0x7fc70414b500 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.050+0000 7fc709d70700 1 --2- 172.21.15.94:0/1674080250 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7040cf860 0x7fc704154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.053 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.050+0000 7fc709d70700 1 --2- 172.21.15.94:0/1674080250 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc7040d0870 0x7fc704156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.050+0000 7fc709d70700 1 -- 172.21.15.94:0/1674080250 >> 172.21.15.94:0/1674080250 conn(0x7fc7040d3ed0 msgr2=0x7fc7040c75e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:38.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.050+0000 7fc709d70700 1 -- 172.21.15.94:0/1674080250 shutdown_connections 2023-11-24T06:07:38.054 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.050+0000 7fc709d70700 1 -- 172.21.15.94:0/1674080250 wait complete. 2023-11-24T06:07:38.066 DEBUG:teuthology.orchestra.run.smithi094:> sudo ceph --cluster ceph osd new 786a3f34-eb17-4f18-8f32-a903286f0a2e 7 2023-11-24T06:07:38.069 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:38.065+0000 7ff0604e1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-24T06:07:38.135 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:38.132+0000 7f17cbd25200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-24T06:07:38.136 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:38.134+0000 7ff0604e1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-24T06:07:38.188 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.185+0000 7f8345dee700 1 Processor -- start 2023-11-24T06:07:38.188 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.185+0000 7f8345dee700 1 -- start start 2023-11-24T06:07:38.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.186+0000 7f8345dee700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f834015a440 0x7f83400cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:38.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.186+0000 7f8345dee700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f834015a820 0x7f83400d04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:38.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.186+0000 7f8345dee700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83400b05f0 con 0x7f834015eb80 2023-11-24T06:07:38.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.186+0000 7f8345dee700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f834015f080 con 0x7f834015a820 2023-11-24T06:07:38.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.186+0000 7f8345dee700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83400afd50 con 0x7f834015a440 2023-11-24T06:07:38.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.186+0000 7f833f7fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f834015a440 0x7f83400cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:38.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.186+0000 7f833f7fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f834015a440 0x7f83400cffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52266/0 (socket says 172.21.15.94:52266) 2023-11-24T06:07:38.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.186+0000 7f833f7fe700 1 -- 172.21.15.94:0/3555752422 learned_addr learned my addr 172.21.15.94:0/3555752422 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-11-24T06:07:38.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.186+0000 7f833effd700 1 --2- 172.21.15.94:0/3555752422 >> v2:172.21.15.91:3300/0 conn(0x7f834015a820 0x7f83400d04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:38.191 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.186+0000 7f833f7fe700 1 -- 172.21.15.94:0/3555752422 >> v2:172.21.15.91:3300/0 conn(0x7f834015a820 msgr2=0x7f83400d04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:38.192 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.186+0000 7f833f7fe700 1 --2- 172.21.15.94:0/3555752422 >> v2:172.21.15.91:3300/0 conn(0x7f834015a820 0x7f83400d04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.192 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.186+0000 7f833f7fe700 1 -- 172.21.15.94:0/3555752422 >> v1:172.21.15.91:6789/0 conn(0x7f834015eb80 legacy=0x7f83400726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:38.192 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.186+0000 7f833f7fe700 1 -- 172.21.15.94:0/3555752422 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83400b0030 con 0x7f834015a440 2023-11-24T06:07:38.192 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.186+0000 7f833f7fe700 1 --2- 172.21.15.94:0/3555752422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f834015a440 0x7f83400cffa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8340069520 tx=0x7f8330009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de3e70fd31e1d63c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:38.192 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.187+0000 7f833e7fc700 1 -- 172.21.15.94:0/3555752422 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8330011830 con 0x7f834015a440 2023-11-24T06:07:38.192 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.187+0000 7f833e7fc700 1 -- 172.21.15.94:0/3555752422 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8330011990 con 0x7f834015a440 2023-11-24T06:07:38.192 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.187+0000 7f833e7fc700 1 -- 172.21.15.94:0/3555752422 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8330011d20 con 0x7f834015a440 2023-11-24T06:07:38.192 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.187+0000 7f8345dee700 1 -- 172.21.15.94:0/3555752422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f834015a440 msgr2=0x7f83400cffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:38.192 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.187+0000 7f8345dee700 1 --2- 172.21.15.94:0/3555752422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f834015a440 0x7f83400cffa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8340069520 tx=0x7f8330009bd0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.192 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.187+0000 7f8345dee700 1 -- 172.21.15.94:0/3555752422 shutdown_connections 2023-11-24T06:07:38.192 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.187+0000 7f8345dee700 1 --2- 172.21.15.94:0/3555752422 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f834015a440 0x7f83400cffa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.192 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.187+0000 7f8345dee700 1 --2- 172.21.15.94:0/3555752422 >> v2:172.21.15.91:3300/0 conn(0x7f834015a820 0x7f83400d04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.192 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.187+0000 7f8345dee700 1 -- 172.21.15.94:0/3555752422 >> 172.21.15.94:0/3555752422 conn(0x7f83400b5070 msgr2=0x7f83400b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:38.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.187+0000 7f8345dee700 1 -- 172.21.15.94:0/3555752422 shutdown_connections 2023-11-24T06:07:38.193 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.187+0000 7f8345dee700 1 -- 172.21.15.94:0/3555752422 wait complete. 2023-11-24T06:07:38.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.188+0000 7f8345dee700 1 Processor -- start 2023-11-24T06:07:38.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.189+0000 7f8345dee700 1 -- start start 2023-11-24T06:07:38.195 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.189+0000 7f8345dee700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f834015a440 0x7f83401506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:38.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.189+0000 7f8345dee700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f834015eb80 0x7f83401e48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:38.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.189+0000 7f8345dee700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83401e4e20 0x7f834006c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:38.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.189+0000 7f8345dee700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83401568f0 con 0x7f834015a440 2023-11-24T06:07:38.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.189+0000 7f8345dee700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83401e5450 con 0x7f834015eb80 2023-11-24T06:07:38.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.189+0000 7f8345dee700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83401e5730 con 0x7f83401e4e20 2023-11-24T06:07:38.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.189+0000 7f833effd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f834015eb80 0x7f83401e48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:38.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f833effd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f834015eb80 0x7f83401e48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.94:52274/0 (socket says 172.21.15.94:52274) 2023-11-24T06:07:38.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f833effd700 1 -- 172.21.15.94:0/131423986 learned_addr learned my addr 172.21.15.94:0/131423986 (peer_addr_for_me v2:172.21.15.94:0/0) 2023-11-24T06:07:38.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f833ffff700 1 --2- 172.21.15.94:0/131423986 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83401e4e20 0x7f834006c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:38.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f833f7fe700 1 --2- 172.21.15.94:0/131423986 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f834015a440 0x7f83401506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:38.196 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f833effd700 1 -- 172.21.15.94:0/131423986 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83401e4e20 msgr2=0x7f834006c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:38.197 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f833effd700 1 --2- 172.21.15.94:0/131423986 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83401e4e20 0x7f834006c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.197 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f833effd700 1 -- 172.21.15.94:0/131423986 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f834015a440 msgr2=0x7f83401506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:38.197 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f833effd700 1 --2- 172.21.15.94:0/131423986 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f834015a440 0x7f83401506d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.197 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f833effd700 1 -- 172.21.15.94:0/131423986 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8330009550 con 0x7f834015eb80 2023-11-24T06:07:38.197 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f833ffff700 1 --2- 172.21.15.94:0/131423986 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83401e4e20 0x7f834006c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:07:38.197 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f833f7fe700 1 --2- 172.21.15.94:0/131423986 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f834015a440 0x7f83401506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:07:38.197 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f833effd700 1 --2- 172.21.15.94:0/131423986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f834015eb80 0x7f83401e48e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f833400e980 tx=0x7f833400ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:38.197 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f8327fff700 1 -- 172.21.15.94:0/131423986 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334009710 con 0x7f834015eb80 2023-11-24T06:07:38.197 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f8327fff700 1 -- 172.21.15.94:0/131423986 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833400c690 con 0x7f834015eb80 2023-11-24T06:07:38.197 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f8327fff700 1 -- 172.21.15.94:0/131423986 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833400ca20 con 0x7f834015eb80 2023-11-24T06:07:38.197 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f8345dee700 1 -- 172.21.15.94:0/131423986 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83401e5a10 con 0x7f834015eb80 2023-11-24T06:07:38.197 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f8345dee700 1 -- 172.21.15.94:0/131423986 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83401e5fd0 con 0x7f834015eb80 2023-11-24T06:07:38.198 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f8327fff700 1 -- 172.21.15.94:0/131423986 <== mon.1 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8334005a20 con 0x7f834015eb80 2023-11-24T06:07:38.198 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.190+0000 7f8327fff700 1 -- 172.21.15.94:0/131423986 <== mon.1 v2:172.21.15.94:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f8334006700 con 0x7f834015eb80 2023-11-24T06:07:38.198 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.191+0000 7f8345dee700 1 -- 172.21.15.94:0/131423986 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83401f3be0 con 0x7f834015eb80 2023-11-24T06:07:38.198 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.193+0000 7f8327fff700 1 -- 172.21.15.94:0/131423986 <== mon.1 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f8334021060 con 0x7f834015eb80 2023-11-24T06:07:38.209 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:38.206+0000 7f17cbd25200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-24T06:07:38.281 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.278+0000 7f8345dee700 1 -- 172.21.15.94:0/131423986 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd new", "uuid": "786a3f34-eb17-4f18-8f32-a903286f0a2e", "id": 7} v 0) v1 -- 0x7f8340075480 con 0x7f834015eb80 2023-11-24T06:07:38.284 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.281+0000 7f8327fff700 1 -- 172.21.15.94:0/131423986 <== mon.1 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "786a3f34-eb17-4f18-8f32-a903286f0a2e", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8334017070 con 0x7f834015eb80 2023-11-24T06:07:38.284 INFO:teuthology.orchestra.run.smithi094.stdout:7 2023-11-24T06:07:38.285 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.281+0000 7f8345dee700 1 -- 172.21.15.94:0/131423986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f834015eb80 msgr2=0x7f83401e48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:38.285 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.281+0000 7f8345dee700 1 --2- 172.21.15.94:0/131423986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f834015eb80 0x7f83401e48e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f833400e980 tx=0x7f833400ec90 comp rx=0 tx=0).stop 2023-11-24T06:07:38.285 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.282+0000 7f8345dee700 1 -- 172.21.15.94:0/131423986 shutdown_connections 2023-11-24T06:07:38.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.282+0000 7f8345dee700 1 --2- 172.21.15.94:0/131423986 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f83401e4e20 0x7f834006c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.282+0000 7f8345dee700 1 --2- 172.21.15.94:0/131423986 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f834015a440 0x7f83401506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.282+0000 7f8345dee700 1 --2- 172.21.15.94:0/131423986 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f834015eb80 0x7f83401e48e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.282+0000 7f8345dee700 1 -- 172.21.15.94:0/131423986 >> 172.21.15.94:0/131423986 conn(0x7f83400b5070 msgr2=0x7f834006e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:38.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.282+0000 7f8345dee700 1 -- 172.21.15.94:0/131423986 shutdown_connections 2023-11-24T06:07:38.286 INFO:teuthology.orchestra.run.smithi094.stderr:2023-11-24T06:07:38.282+0000 7f8345dee700 1 -- 172.21.15.94:0/131423986 wait complete. 2023-11-24T06:07:38.297 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-24T06:07:38.297 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-11-24T06:07:38.299 INFO:tasks.ceph.osd.0:Started 2023-11-24T06:07:38.299 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-24T06:07:38.299 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-11-24T06:07:38.301 INFO:tasks.ceph.osd.1:Started 2023-11-24T06:07:38.301 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-24T06:07:38.301 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-11-24T06:07:38.303 INFO:tasks.ceph.osd.2:Started 2023-11-24T06:07:38.303 INFO:tasks.ceph.osd.3:Restarting daemon 2023-11-24T06:07:38.303 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-11-24T06:07:38.305 INFO:tasks.ceph.osd.3:Started 2023-11-24T06:07:38.305 INFO:tasks.ceph.osd.4:Restarting daemon 2023-11-24T06:07:38.305 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-11-24T06:07:38.307 INFO:tasks.ceph.osd.4:Started 2023-11-24T06:07:38.307 INFO:tasks.ceph.osd.5:Restarting daemon 2023-11-24T06:07:38.308 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-11-24T06:07:38.310 INFO:tasks.ceph.osd.5:Started 2023-11-24T06:07:38.310 INFO:tasks.ceph.osd.6:Restarting daemon 2023-11-24T06:07:38.310 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-11-24T06:07:38.312 INFO:tasks.ceph.osd.6:Started 2023-11-24T06:07:38.312 INFO:tasks.ceph.osd.7:Restarting daemon 2023-11-24T06:07:38.312 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-11-24T06:07:38.314 INFO:tasks.ceph.osd.7:Started 2023-11-24T06:07:38.315 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:07:38.460 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.457+0000 7f9567d72700 1 Processor -- start 2023-11-24T06:07:38.460 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.457+0000 7f9567d72700 1 -- start start 2023-11-24T06:07:38.462 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.458+0000 7f9567d72700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f9560143ae0 0x7f95600b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:38.463 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.458+0000 7f9567d72700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f95601440f0 0x7f9560071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:38.463 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.458+0000 7f9567d72700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9560143ec0 con 0x7f95600720f0 2023-11-24T06:07:38.463 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.458+0000 7f9567d72700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95600b40a0 con 0x7f9560143ae0 2023-11-24T06:07:38.463 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.458+0000 7f9567d72700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9560072580 con 0x7f95601440f0 2023-11-24T06:07:38.463 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.458+0000 7f9565b0e700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f9560143ae0 0x7f95600b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:38.463 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.458+0000 7f956630f700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f95600720f0 0x7f956015e8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52034/0 (socket says 172.21.15.91:52034) 2023-11-24T06:07:38.463 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.458+0000 7f956630f700 1 -- 172.21.15.91:0/1220191944 learned_addr learned my addr 172.21.15.91:0/1220191944 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:07:38.463 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.458+0000 7f956530d700 1 --2- 172.21.15.91:0/1220191944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f95601440f0 0x7f9560071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:38.463 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.458+0000 7f9564b0c700 1 -- 172.21.15.91:0/1220191944 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118004588 0 0) 0x7f9560143ec0 con 0x7f95600720f0 2023-11-24T06:07:38.463 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.458+0000 7f9564b0c700 1 -- 172.21.15.91:0/1220191944 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f954c0035b0 con 0x7f95600720f0 2023-11-24T06:07:38.463 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.459+0000 7f9565b0e700 1 -- 172.21.15.91:0/1220191944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f95601440f0 msgr2=0x7f9560071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:38.463 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.459+0000 7f9565b0e700 1 --2- 172.21.15.91:0/1220191944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f95601440f0 0x7f9560071bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.464 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.459+0000 7f9565b0e700 1 -- 172.21.15.91:0/1220191944 >> v1:172.21.15.91:6789/0 conn(0x7f95600720f0 legacy=0x7f956015e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:38.464 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.459+0000 7f9565b0e700 1 -- 172.21.15.91:0/1220191944 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95600b3ae0 con 0x7f9560143ae0 2023-11-24T06:07:38.464 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.459+0000 7f9565b0e700 1 --2- 172.21.15.91:0/1220191944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9560143ae0 0x7f95600b51b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f955c009840 tx=0x7f955c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65e4e28f9445b1b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:38.464 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.459+0000 7f9564b0c700 1 -- 172.21.15.91:0/1220191944 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c011830 con 0x7f9560143ae0 2023-11-24T06:07:38.464 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.459+0000 7f9564b0c700 1 -- 172.21.15.91:0/1220191944 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955c011990 con 0x7f9560143ae0 2023-11-24T06:07:38.464 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.459+0000 7f9564b0c700 1 -- 172.21.15.91:0/1220191944 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c011d20 con 0x7f9560143ae0 2023-11-24T06:07:38.464 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.459+0000 7f9567d72700 1 -- 172.21.15.91:0/1220191944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9560143ae0 msgr2=0x7f95600b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:38.464 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.459+0000 7f9567d72700 1 --2- 172.21.15.91:0/1220191944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9560143ae0 0x7f95600b51b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f955c009840 tx=0x7f955c009b50 comp rx=0 tx=0).stop 2023-11-24T06:07:38.464 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.459+0000 7f9567d72700 1 -- 172.21.15.91:0/1220191944 shutdown_connections 2023-11-24T06:07:38.464 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.459+0000 7f9567d72700 1 --2- 172.21.15.91:0/1220191944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9560143ae0 0x7f95600b51b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.464 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.459+0000 7f9567d72700 1 --2- 172.21.15.91:0/1220191944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f95601440f0 0x7f9560071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.464 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.459+0000 7f9567d72700 1 -- 172.21.15.91:0/1220191944 >> 172.21.15.91:0/1220191944 conn(0x7f95600b2740 msgr2=0x7f95600b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:38.464 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.460+0000 7f9567d72700 1 -- 172.21.15.91:0/1220191944 shutdown_connections 2023-11-24T06:07:38.465 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.460+0000 7f9567d72700 1 -- 172.21.15.91:0/1220191944 wait complete. 2023-11-24T06:07:38.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.461+0000 7f9567d72700 1 Processor -- start 2023-11-24T06:07:38.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.461+0000 7f9567d72700 1 -- start start 2023-11-24T06:07:38.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.461+0000 7f9567d72700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f95600720f0 0x7f956013e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:38.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.461+0000 7f9567d72700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9560143ae0 0x7f956006eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:38.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.461+0000 7f9567d72700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95601440f0 0x7f956006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:38.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.461+0000 7f9567d72700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956013eb80 con 0x7f95601440f0 2023-11-24T06:07:38.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.461+0000 7f9567d72700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9560159760 con 0x7f9560143ae0 2023-11-24T06:07:38.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.461+0000 7f9567d72700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9560159a40 con 0x7f95600720f0 2023-11-24T06:07:38.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.462+0000 7f956630f700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95601440f0 0x7f956006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:38.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.462+0000 7f956630f700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95601440f0 0x7f956006f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57106/0 (socket says 172.21.15.91:57106) 2023-11-24T06:07:38.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.462+0000 7f956630f700 1 -- 172.21.15.91:0/923097395 learned_addr learned my addr 172.21.15.91:0/923097395 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:38.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.462+0000 7f9565b0e700 1 --2- 172.21.15.91:0/923097395 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f95600720f0 0x7f956013e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:38.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.462+0000 7f956530d700 1 --2- 172.21.15.91:0/923097395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9560143ae0 0x7f956006eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:38.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.462+0000 7f956630f700 1 -- 172.21.15.91:0/923097395 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f95600720f0 msgr2=0x7f956013e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:38.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.462+0000 7f956630f700 1 --2- 172.21.15.91:0/923097395 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f95600720f0 0x7f956013e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.462+0000 7f956630f700 1 -- 172.21.15.91:0/923097395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9560143ae0 msgr2=0x7f956006eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:38.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.462+0000 7f956630f700 1 --2- 172.21.15.91:0/923097395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9560143ae0 0x7f956006eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.462+0000 7f956630f700 1 -- 172.21.15.91:0/923097395 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955c0094a0 con 0x7f95601440f0 2023-11-24T06:07:38.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.462+0000 7f956630f700 1 --2- 172.21.15.91:0/923097395 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95601440f0 0x7f956006f080 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f955800caa0 tx=0x7f955800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:38.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.462+0000 7f95567fc700 1 -- 172.21.15.91:0/923097395 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955800a9e0 con 0x7f95601440f0 2023-11-24T06:07:38.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.462+0000 7f95567fc700 1 -- 172.21.15.91:0/923097395 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955800ab40 con 0x7f95601440f0 2023-11-24T06:07:38.469 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.462+0000 7f95567fc700 1 -- 172.21.15.91:0/923097395 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9558012400 con 0x7f95601440f0 2023-11-24T06:07:38.469 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.462+0000 7f9567d72700 1 -- 172.21.15.91:0/923097395 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9560159d20 con 0x7f95601440f0 2023-11-24T06:07:38.469 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.462+0000 7f9567d72700 1 -- 172.21.15.91:0/923097395 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956015a2e0 con 0x7f95601440f0 2023-11-24T06:07:38.469 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.463+0000 7f95567fc700 1 -- 172.21.15.91:0/923097395 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9558012560 con 0x7f95601440f0 2023-11-24T06:07:38.469 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.463+0000 7f95567fc700 1 -- 172.21.15.91:0/923097395 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f955801ba80 con 0x7f95601440f0 2023-11-24T06:07:38.469 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.463+0000 7f9567d72700 1 -- 172.21.15.91:0/923097395 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95601447e0 con 0x7f95601440f0 2023-11-24T06:07:38.469 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.465+0000 7f95567fc700 1 -- 172.21.15.91:0/923097395 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f9558017070 con 0x7f95601440f0 2023-11-24T06:07:38.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.536+0000 7f9567d72700 1 -- 172.21.15.91:0/923097395 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9560068670 con 0x7f95601440f0 2023-11-24T06:07:38.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.536+0000 7f95567fc700 1 -- 172.21.15.91:0/923097395 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f955801bc90 con 0x7f95601440f0 2023-11-24T06:07:38.539 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:38.540 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":10,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:07:38.279996+0000","last_up_change":"0.000000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:07:38.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.537+0000 7f9567d72700 1 -- 172.21.15.91:0/923097395 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95601440f0 msgr2=0x7f956006f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:38.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.537+0000 7f9567d72700 1 --2- 172.21.15.91:0/923097395 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95601440f0 0x7f956006f080 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f955800caa0 tx=0x7f955800cdb0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.537+0000 7f9567d72700 1 -- 172.21.15.91:0/923097395 shutdown_connections 2023-11-24T06:07:38.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.537+0000 7f9567d72700 1 --2- 172.21.15.91:0/923097395 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95601440f0 0x7f956006f080 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.537+0000 7f9567d72700 1 --2- 172.21.15.91:0/923097395 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f95600720f0 0x7f956013e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.537+0000 7f9567d72700 1 --2- 172.21.15.91:0/923097395 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9560143ae0 0x7f956006eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:38.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.537+0000 7f9567d72700 1 -- 172.21.15.91:0/923097395 >> 172.21.15.91:0/923097395 conn(0x7f95600b2740 msgr2=0x7f95600cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:38.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.537+0000 7f9567d72700 1 -- 172.21.15.91:0/923097395 shutdown_connections 2023-11-24T06:07:38.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:38.537+0000 7f9567d72700 1 -- 172.21.15.91:0/923097395 wait complete. 2023-11-24T06:07:38.552 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-24T06:07:38.552 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-24T06:07:38.875 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:38.872+0000 7ff0604e1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-24T06:07:38.911 INFO:tasks.ceph.osd.6.smithi094.stderr:2023-11-24T06:07:38.908+0000 7f500501c640 -1 Falling back to public interface 2023-11-24T06:07:38.916 INFO:tasks.ceph.osd.5.smithi094.stderr:2023-11-24T06:07:38.913+0000 7fbabd5ad640 -1 Falling back to public interface 2023-11-24T06:07:38.919 INFO:tasks.ceph.osd.3.smithi091.stderr:2023-11-24T06:07:38.916+0000 7f11a7d70640 -1 Falling back to public interface 2023-11-24T06:07:38.920 INFO:tasks.ceph.osd.7.smithi094.stderr:2023-11-24T06:07:38.917+0000 7f6ed9d99640 -1 Falling back to public interface 2023-11-24T06:07:38.934 INFO:tasks.ceph.osd.2.smithi091.stderr:2023-11-24T06:07:38.931+0000 7f621438f640 -1 Falling back to public interface 2023-11-24T06:07:38.936 INFO:tasks.ceph.osd.1.smithi091.stderr:2023-11-24T06:07:38.933+0000 7f2f6ba33640 -1 Falling back to public interface 2023-11-24T06:07:38.936 INFO:tasks.ceph.osd.0.smithi091.stderr:2023-11-24T06:07:38.934+0000 7f5968cf2640 -1 Falling back to public interface 2023-11-24T06:07:38.937 INFO:tasks.ceph.osd.2.smithi091.stderr:2023-11-24T06:07:38.934+0000 7f620b74d700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6813/0: (98) Address already in use 2023-11-24T06:07:38.938 INFO:tasks.ceph.osd.4.smithi094.stderr:2023-11-24T06:07:38.935+0000 7febf30cf640 -1 Falling back to public interface 2023-11-24T06:07:38.954 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:38.951+0000 7f17cbd25200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-24T06:07:38.957 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:38.954+0000 7ff0604e1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-24T06:07:39.032 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:39.029+0000 7f17cbd25200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-24T06:07:39.097 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:39.094+0000 7ff0604e1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-24T06:07:39.173 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:39.170+0000 7f17cbd25200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-24T06:07:39.332 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:39.329+0000 7ff0604e1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-24T06:07:39.356 DEBUG:teuthology.orchestra.run.smithi091:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-24T06:07:39.410 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:39.407+0000 7f17cbd25200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-24T06:07:39.472 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.469+0000 7f42f3d86700 1 Processor -- start 2023-11-24T06:07:39.472 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.469+0000 7f42f3d86700 1 -- start start 2023-11-24T06:07:39.477 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.470+0000 7f42f3d86700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42ec0cf2b0 0x7f42ec15c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:39.477 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.470+0000 7f42f3d86700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42ec0d0a90 con 0x7f42ec15bde0 2023-11-24T06:07:39.477 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.470+0000 7f42f3d86700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42ec06ee10 con 0x7f42ec0ce5c0 2023-11-24T06:07:39.477 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.470+0000 7f42f3d86700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec06f0f0 con 0x7f42ec0cf2b0 2023-11-24T06:07:39.477 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.470+0000 7f42f1b22700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f42ec15bde0 0x7f42ec0d01b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52042/0 (socket says 172.21.15.91:52042) 2023-11-24T06:07:39.477 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.470+0000 7f42f1b22700 1 -- 172.21.15.91:0/3131309299 learned_addr learned my addr 172.21.15.91:0/3131309299 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:07:39.477 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.470+0000 7f42f1321700 1 --2- 172.21.15.91:0/3131309299 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42ec0cf2b0 0x7f42ec15c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:39.477 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 -- 172.21.15.91:0/3131309299 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074147109 0 0) 0x7f42ec0d0a90 con 0x7f42ec15bde0 2023-11-24T06:07:39.477 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 -- 172.21.15.91:0/3131309299 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42d80035b0 con 0x7f42ec15bde0 2023-11-24T06:07:39.478 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 -- 172.21.15.91:0/3131309299 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400596972 0 0) 0x7f42ec06ee10 con 0x7f42ec0ce5c0 2023-11-24T06:07:39.478 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 -- 172.21.15.91:0/3131309299 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42ec0d0a90 con 0x7f42ec0ce5c0 2023-11-24T06:07:39.478 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 -- 172.21.15.91:0/3131309299 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2727103721 0 0) 0x7f42d80035b0 con 0x7f42ec15bde0 2023-11-24T06:07:39.478 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 -- 172.21.15.91:0/3131309299 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42ec0cf2b0 msgr2=0x7f42ec15c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:39.478 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 --2- 172.21.15.91:0/3131309299 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42ec0cf2b0 0x7f42ec15c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:39.478 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 -- 172.21.15.91:0/3131309299 >> v1:172.21.15.94:6789/0 conn(0x7f42ec0ce5c0 legacy=0x7f42ec06fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:39.478 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 -- 172.21.15.91:0/3131309299 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec06f3d0 con 0x7f42ec15bde0 2023-11-24T06:07:39.478 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 -- 172.21.15.91:0/3131309299 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f42e80019f0 con 0x7f42ec15bde0 2023-11-24T06:07:39.478 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 -- 172.21.15.91:0/3131309299 >> v1:172.21.15.91:6789/0 conn(0x7f42ec15bde0 legacy=0x7f42ec0d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:39.478 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 --2- 172.21.15.91:0/3131309299 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42d8003bc0 0x7f42d8003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:39.478 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 --2- 172.21.15.91:0/3131309299 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42d8004850 0x7f42d8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:39.478 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 --2- 172.21.15.91:0/3131309299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d8005210 0x7f42d800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:39.479 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 -- 172.21.15.91:0/3131309299 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec06f0f0 con 0x7f42d8003bc0 2023-11-24T06:07:39.479 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 -- 172.21.15.91:0/3131309299 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d800c080 con 0x7f42d8005210 2023-11-24T06:07:39.479 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f0b20700 1 -- 172.21.15.91:0/3131309299 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d800c240 con 0x7f42d8004850 2023-11-24T06:07:39.479 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f1321700 1 --2- 172.21.15.91:0/3131309299 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42ec0cf2b0 0x7f42ec15c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:07:39.479 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f1b22700 1 --2- 172.21.15.91:0/3131309299 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42d8003bc0 0x7f42d8003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:39.479 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f2323700 1 --2- 172.21.15.91:0/3131309299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d8005210 0x7f42d800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:39.479 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f1321700 1 --2- 172.21.15.91:0/3131309299 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42d8004850 0x7f42d8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:39.479 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f1b22700 1 -- 172.21.15.91:0/3131309299 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42d8004850 msgr2=0x7f42d8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:39.479 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f1b22700 1 --2- 172.21.15.91:0/3131309299 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42d8004850 0x7f42d8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:39.479 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f1b22700 1 -- 172.21.15.91:0/3131309299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d8005210 msgr2=0x7f42d800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:39.479 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f1b22700 1 --2- 172.21.15.91:0/3131309299 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d8005210 0x7f42d800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:39.479 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f1b22700 1 -- 172.21.15.91:0/3131309299 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d80035b0 con 0x7f42d8003bc0 2023-11-24T06:07:39.480 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.471+0000 7f42f1321700 1 --2- 172.21.15.91:0/3131309299 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42d8004850 0x7f42d8004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:07:39.480 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.472+0000 7f42f1b22700 1 --2- 172.21.15.91:0/3131309299 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42d8003bc0 0x7f42d8003fc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f42e800a7c0 tx=0x7f42d8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0c0711ed1ba405 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:39.480 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.472+0000 7f42f0b20700 1 -- 172.21.15.91:0/3131309299 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e8012600 con 0x7f42d8003bc0 2023-11-24T06:07:39.480 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.472+0000 7f42f0b20700 1 -- 172.21.15.91:0/3131309299 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e8012780 con 0x7f42d8003bc0 2023-11-24T06:07:39.480 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.472+0000 7f42f0b20700 1 -- 172.21.15.91:0/3131309299 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e8012b30 con 0x7f42d8003bc0 2023-11-24T06:07:39.480 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.472+0000 7f42f3d86700 1 -- 172.21.15.91:0/3131309299 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42d8003bc0 msgr2=0x7f42d8003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:39.480 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.472+0000 7f42f3d86700 1 --2- 172.21.15.91:0/3131309299 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42d8003bc0 0x7f42d8003fc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f42e800a7c0 tx=0x7f42d8005880 comp rx=0 tx=0).stop 2023-11-24T06:07:39.480 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.472+0000 7f42f2323700 1 -- 172.21.15.91:0/3131309299 reap_dead start 2023-11-24T06:07:39.480 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.472+0000 7f42f3d86700 1 -- 172.21.15.91:0/3131309299 shutdown_connections 2023-11-24T06:07:39.480 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.472+0000 7f42f3d86700 1 -- 172.21.15.91:0/3131309299 >> 172.21.15.91:0/3131309299 conn(0x7f42ec0b3720 msgr2=0x7f42ec0b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:39.481 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.472+0000 7f42f3d86700 1 -- 172.21.15.91:0/3131309299 shutdown_connections 2023-11-24T06:07:39.481 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.473+0000 7f42f3d86700 1 -- 172.21.15.91:0/3131309299 wait complete. 2023-11-24T06:07:39.481 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.474+0000 7f42f3d86700 1 Processor -- start 2023-11-24T06:07:39.481 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.474+0000 7f42f3d86700 1 -- start start 2023-11-24T06:07:39.483 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.474+0000 7f42f3d86700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d8003bc0 0x7f42ec0c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:39.484 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.474+0000 7f42f3d86700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42ec0b6c00 0x7f42ec0b5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:39.484 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.474+0000 7f42f3d86700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec0b5d90 0x7f42ec0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:39.484 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.474+0000 7f42f3d86700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec0b6780 con 0x7f42ec0b5d90 2023-11-24T06:07:39.484 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.474+0000 7f42f3d86700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec0ca5e0 con 0x7f42d8003bc0 2023-11-24T06:07:39.484 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.474+0000 7f42f3d86700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec0ca020 con 0x7f42ec0b6c00 2023-11-24T06:07:39.484 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42f2323700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec0b5d90 0x7f42ec0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:39.484 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42f2323700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec0b5d90 0x7f42ec0b6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57124/0 (socket says 172.21.15.91:57124) 2023-11-24T06:07:39.484 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42f1321700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42ec0b6c00 0x7f42ec0b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:39.484 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42f2323700 1 -- 172.21.15.91:0/3593842841 learned_addr learned my addr 172.21.15.91:0/3593842841 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:39.484 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42f1b22700 1 --2- 172.21.15.91:0/3593842841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d8003bc0 0x7f42ec0c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:39.484 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42f2323700 1 -- 172.21.15.91:0/3593842841 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42ec0b6c00 msgr2=0x7f42ec0b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:39.484 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42f2323700 1 --2- 172.21.15.91:0/3593842841 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42ec0b6c00 0x7f42ec0b5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:39.484 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42f2323700 1 -- 172.21.15.91:0/3593842841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d8003bc0 msgr2=0x7f42ec0c96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:39.485 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42f2323700 1 --2- 172.21.15.91:0/3593842841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d8003bc0 0x7f42ec0c96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:39.485 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42f2323700 1 -- 172.21.15.91:0/3593842841 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec0ca300 con 0x7f42ec0b5d90 2023-11-24T06:07:39.485 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42f2323700 1 --2- 172.21.15.91:0/3593842841 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec0b5d90 0x7f42ec0b6240 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0c8c30 tx=0x7f42e400fa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:39.485 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42e27fc700 1 -- 172.21.15.91:0/3593842841 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e40052e0 con 0x7f42ec0b5d90 2023-11-24T06:07:39.485 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42e27fc700 1 -- 172.21.15.91:0/3593842841 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e4005460 con 0x7f42ec0b5d90 2023-11-24T06:07:39.485 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42e27fc700 1 -- 172.21.15.91:0/3593842841 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e40038c0 con 0x7f42ec0b5d90 2023-11-24T06:07:39.485 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42f3d86700 1 -- 172.21.15.91:0/3593842841 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42ec15c6d0 con 0x7f42ec0b5d90 2023-11-24T06:07:39.485 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42f3d86700 1 -- 172.21.15.91:0/3593842841 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42ec0c9bf0 con 0x7f42ec0b5d90 2023-11-24T06:07:39.485 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42e27fc700 1 -- 172.21.15.91:0/3593842841 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f42e4003d20 con 0x7f42ec0b5d90 2023-11-24T06:07:39.485 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.475+0000 7f42e27fc700 1 -- 172.21.15.91:0/3593842841 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f42e401dbe0 con 0x7f42ec0b5d90 2023-11-24T06:07:39.485 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.476+0000 7f42f3d86700 1 -- 172.21.15.91:0/3593842841 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42ec0c9db0 con 0x7f42ec0b5d90 2023-11-24T06:07:39.485 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.479+0000 7f42e27fc700 1 -- 172.21.15.91:0/3593842841 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f42e401d510 con 0x7f42ec0b5d90 2023-11-24T06:07:39.548 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:39.545+0000 7ff0604e1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-24T06:07:39.553 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.551+0000 7f42f3d86700 1 -- 172.21.15.91:0/3593842841 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42ec0b4040 con 0x7f42ec0b5d90 2023-11-24T06:07:39.554 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.551+0000 7f42e27fc700 1 -- 172.21.15.91:0/3593842841 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f42e402c090 con 0x7f42ec0b5d90 2023-11-24T06:07:39.554 INFO:teuthology.misc.health.smithi091.stdout: 2023-11-24T06:07:39.554 INFO:teuthology.misc.health.smithi091.stdout:{"epoch":10,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:07:38.279996+0000","last_up_change":"0.000000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:07:39.555 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.552+0000 7f42f3d86700 1 -- 172.21.15.91:0/3593842841 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec0b5d90 msgr2=0x7f42ec0b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:39.555 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.552+0000 7f42f3d86700 1 --2- 172.21.15.91:0/3593842841 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec0b5d90 0x7f42ec0b6240 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0c8c30 tx=0x7f42e400fa50 comp rx=0 tx=0).stop 2023-11-24T06:07:39.555 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.552+0000 7f42f3d86700 1 -- 172.21.15.91:0/3593842841 shutdown_connections 2023-11-24T06:07:39.555 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.552+0000 7f42f3d86700 1 --2- 172.21.15.91:0/3593842841 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42d8003bc0 0x7f42ec0c96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:39.555 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.552+0000 7f42f3d86700 1 --2- 172.21.15.91:0/3593842841 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec0b5d90 0x7f42ec0b6240 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:39.556 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.552+0000 7f42f3d86700 1 --2- 172.21.15.91:0/3593842841 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42ec0b6c00 0x7f42ec0b5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:39.556 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.552+0000 7f42f3d86700 1 -- 172.21.15.91:0/3593842841 >> 172.21.15.91:0/3593842841 conn(0x7f42ec0b3720 msgr2=0x7f42ec0b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:39.556 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.552+0000 7f42f3d86700 1 -- 172.21.15.91:0/3593842841 shutdown_connections 2023-11-24T06:07:39.556 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:39.552+0000 7f42f3d86700 1 -- 172.21.15.91:0/3593842841 wait complete. 2023-11-24T06:07:39.566 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-24T06:07:39.620 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:39.617+0000 7f17cbd25200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-24T06:07:39.682 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:39.679+0000 7ff0604e1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-24T06:07:39.754 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:39.750+0000 7f17cbd25200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-24T06:07:39.902 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:39.900+0000 7ff0604e1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-24T06:07:39.972 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:39.969+0000 7f17cbd25200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-24T06:07:40.059 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:40.056+0000 7ff0604e1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-24T06:07:40.128 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:40.124+0000 7f17cbd25200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-24T06:07:40.170 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:40.167+0000 7ff0604e1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-24T06:07:40.241 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:40.238+0000 7f17cbd25200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-24T06:07:40.490 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:40.487+0000 7ff0604e1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-24T06:07:40.562 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:40.559+0000 7f17cbd25200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-24T06:07:40.589 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:40.586+0000 7ff0604e1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-24T06:07:40.662 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:40.659+0000 7f17cbd25200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-24T06:07:40.776 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:40.774+0000 7ff0604e1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-24T06:07:40.849 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:40.846+0000 7f17cbd25200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-24T06:07:41.374 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:41.371+0000 7ff0604e1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-24T06:07:41.458 INFO:tasks.ceph.mgr.x.smithi094.stderr:2023-11-24T06:07:41.455+0000 7f17cbd25200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-24T06:07:42.397 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:42.394+0000 7ff018436700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-11-24T06:07:42.398 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:07:42.395+0000 7ff018436700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-11-24T06:07:42.405 INFO:tasks.ceph.osd.7.smithi094.stderr:2023-11-24T06:07:42.402+0000 7f6ed9d99640 -1 osd.7 0 log_to_monitors true 2023-11-24T06:07:42.442 INFO:tasks.ceph.osd.5.smithi094.stderr:2023-11-24T06:07:42.439+0000 7fbabd5ad640 -1 osd.5 0 log_to_monitors true 2023-11-24T06:07:42.449 INFO:tasks.ceph.osd.3.smithi091.stderr:2023-11-24T06:07:42.446+0000 7f11a7d70640 -1 osd.3 0 log_to_monitors true 2023-11-24T06:07:42.449 INFO:tasks.ceph.osd.4.smithi094.stderr:2023-11-24T06:07:42.446+0000 7febf30cf640 -1 osd.4 0 log_to_monitors true 2023-11-24T06:07:42.454 INFO:tasks.ceph.osd.6.smithi094.stderr:2023-11-24T06:07:42.451+0000 7f500501c640 -1 osd.6 0 log_to_monitors true 2023-11-24T06:07:42.482 INFO:tasks.ceph.osd.0.smithi091.stderr:2023-11-24T06:07:42.479+0000 7f5968cf2640 -1 osd.0 0 log_to_monitors true 2023-11-24T06:07:42.488 INFO:tasks.ceph.osd.1.smithi091.stderr:2023-11-24T06:07:42.486+0000 7f2f6ba33640 -1 osd.1 0 log_to_monitors true 2023-11-24T06:07:42.489 INFO:tasks.ceph.osd.2.smithi091.stderr:2023-11-24T06:07:42.486+0000 7f621438f640 -1 osd.2 0 log_to_monitors true 2023-11-24T06:07:45.737 INFO:tasks.ceph.osd.5.smithi094.stderr:2023-11-24T06:07:45.733+0000 7fbab1164700 -1 osd.5 0 waiting for initial osdmap 2023-11-24T06:07:45.751 INFO:tasks.ceph.osd.5.smithi094.stderr:2023-11-24T06:07:45.748+0000 7fbaa9544700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T06:07:45.760 INFO:tasks.ceph.osd.4.smithi094.stderr:2023-11-24T06:07:45.757+0000 7febe6c86700 -1 osd.4 0 waiting for initial osdmap 2023-11-24T06:07:45.762 INFO:tasks.ceph.osd.6.smithi094.stderr:2023-11-24T06:07:45.759+0000 7f4ff8bd3700 -1 osd.6 0 waiting for initial osdmap 2023-11-24T06:07:45.774 INFO:tasks.ceph.osd.6.smithi094.stderr:2023-11-24T06:07:45.771+0000 7f4ff0fb3700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T06:07:45.774 INFO:tasks.ceph.osd.4.smithi094.stderr:2023-11-24T06:07:45.771+0000 7febdf066700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T06:07:45.808 INFO:tasks.ceph.osd.7.smithi094.stderr:2023-11-24T06:07:45.805+0000 7f6ecd950700 -1 osd.7 0 waiting for initial osdmap 2023-11-24T06:07:45.818 INFO:tasks.ceph.osd.7.smithi094.stderr:2023-11-24T06:07:45.815+0000 7f6ec5d30700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T06:07:46.373 DEBUG:teuthology.orchestra.run.smithi091:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-24T06:07:46.441 INFO:tasks.ceph.osd.3.smithi091.stderr:2023-11-24T06:07:46.438+0000 7f119b927700 -1 osd.3 0 waiting for initial osdmap 2023-11-24T06:07:46.456 INFO:tasks.ceph.osd.3.smithi091.stderr:2023-11-24T06:07:46.450+0000 7f1193d07700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T06:07:46.468 INFO:tasks.ceph.osd.0.smithi091.stderr:2023-11-24T06:07:46.465+0000 7f595c8a9700 -1 osd.0 0 waiting for initial osdmap 2023-11-24T06:07:46.477 INFO:tasks.ceph.osd.1.smithi091.stderr:2023-11-24T06:07:46.474+0000 7f2f5f5ea700 -1 osd.1 0 waiting for initial osdmap 2023-11-24T06:07:46.481 INFO:tasks.ceph.osd.0.smithi091.stderr:2023-11-24T06:07:46.479+0000 7f5954c89700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T06:07:46.490 INFO:tasks.ceph.osd.2.smithi091.stderr:2023-11-24T06:07:46.487+0000 7f6207f46700 -1 osd.2 0 waiting for initial osdmap 2023-11-24T06:07:46.493 INFO:tasks.ceph.osd.1.smithi091.stderr:2023-11-24T06:07:46.490+0000 7f2f579ca700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T06:07:46.502 INFO:tasks.ceph.osd.2.smithi091.stderr:2023-11-24T06:07:46.500+0000 7f6200326700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-24T06:07:46.523 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.520+0000 7f54f1092700 1 Processor -- start 2023-11-24T06:07:46.523 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.520+0000 7f54f1092700 1 -- start start 2023-11-24T06:07:46.525 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.521+0000 7f54f1092700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f54ec0cf2b0 0x7f54ec15c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:46.525 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.521+0000 7f54f1092700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54ec0d0a90 con 0x7f54ec15bde0 2023-11-24T06:07:46.525 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.521+0000 7f54f1092700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54ec06ee10 con 0x7f54ec0ce5c0 2023-11-24T06:07:46.526 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.521+0000 7f54f1092700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54ec06f0f0 con 0x7f54ec0cf2b0 2023-11-24T06:07:46.526 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.521+0000 7f54ea59c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f54ec0cf2b0 0x7f54ec15c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:46.526 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.521+0000 7f54ea59c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f54ec0cf2b0 0x7f54ec15c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57226/0 (socket says 172.21.15.91:57226) 2023-11-24T06:07:46.526 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.521+0000 7f54ead9d700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f54ec15bde0 0x7f54ec0d01b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52158/0 (socket says 172.21.15.91:52158) 2023-11-24T06:07:46.526 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.521+0000 7f54ea59c700 1 -- 172.21.15.91:0/2249755332 learned_addr learned my addr 172.21.15.91:0/2249755332 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:46.526 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.521+0000 7f54ea59c700 1 -- 172.21.15.91:0/2249755332 >> v1:172.21.15.94:6789/0 conn(0x7f54ec0ce5c0 legacy=0x7f54ec06fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:46.526 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.521+0000 7f54ea59c700 1 -- 172.21.15.91:0/2249755332 >> v1:172.21.15.91:6789/0 conn(0x7f54ec15bde0 legacy=0x7f54ec0d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:46.526 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.521+0000 7f54ea59c700 1 -- 172.21.15.91:0/2249755332 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ec06f3d0 con 0x7f54ec0cf2b0 2023-11-24T06:07:46.526 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.521+0000 7f54e9d9b700 1 -- 172.21.15.91:0/2249755332 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073237062 0 0) 0x7f54ec0d0a90 con 0x7f54ec15bde0 2023-11-24T06:07:46.526 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.522+0000 7f54ea59c700 1 --2- 172.21.15.91:0/2249755332 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f54ec0cf2b0 0x7f54ec15c1c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f54d40098f0 tx=0x7f54d4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65bc2f7eb4eb00e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:46.526 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.522+0000 7f54e9d9b700 1 -- 172.21.15.91:0/2249755332 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d400b8a0 con 0x7f54ec0cf2b0 2023-11-24T06:07:46.526 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.522+0000 7f54e9d9b700 1 -- 172.21.15.91:0/2249755332 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d400ba00 con 0x7f54ec0cf2b0 2023-11-24T06:07:46.526 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.522+0000 7f54e9d9b700 1 -- 172.21.15.91:0/2249755332 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d400bd90 con 0x7f54ec0cf2b0 2023-11-24T06:07:46.527 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.522+0000 7f54f1092700 1 -- 172.21.15.91:0/2249755332 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f54ec0cf2b0 msgr2=0x7f54ec15c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:46.527 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.522+0000 7f54f1092700 1 --2- 172.21.15.91:0/2249755332 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f54ec0cf2b0 0x7f54ec15c1c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f54d40098f0 tx=0x7f54d4009c00 comp rx=0 tx=0).stop 2023-11-24T06:07:46.527 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.522+0000 7f54f1092700 1 -- 172.21.15.91:0/2249755332 shutdown_connections 2023-11-24T06:07:46.527 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.522+0000 7f54f1092700 1 --2- 172.21.15.91:0/2249755332 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f54ec0cf2b0 0x7f54ec15c1c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:46.527 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.522+0000 7f54f1092700 1 -- 172.21.15.91:0/2249755332 >> 172.21.15.91:0/2249755332 conn(0x7f54ec0b3720 msgr2=0x7f54ec0b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:46.527 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.522+0000 7f54f1092700 1 -- 172.21.15.91:0/2249755332 shutdown_connections 2023-11-24T06:07:46.527 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.522+0000 7f54f1092700 1 -- 172.21.15.91:0/2249755332 wait complete. 2023-11-24T06:07:46.530 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.524+0000 7f54f1092700 1 Processor -- start 2023-11-24T06:07:46.530 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.524+0000 7f54f1092700 1 -- start start 2023-11-24T06:07:46.530 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.524+0000 7f54f1092700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f54ec0ce5c0 0x7f54ec0b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:46.530 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.524+0000 7f54f1092700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f54ec0cf2b0 0x7f54ec0b6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:46.530 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.524+0000 7f54f1092700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f54ec15bde0 0x7f54ec14ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:46.530 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.524+0000 7f54f1092700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ec1f19e0 con 0x7f54ec0ce5c0 2023-11-24T06:07:46.530 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.524+0000 7f54f1092700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54ec0b2860 con 0x7f54ec0cf2b0 2023-11-24T06:07:46.530 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.524+0000 7f54f1092700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54ec15c260 con 0x7f54ec15bde0 2023-11-24T06:07:46.530 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54ead9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f54ec0ce5c0 0x7f54ec0b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:46.530 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54ead9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f54ec0ce5c0 0x7f54ec0b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57230/0 (socket says 172.21.15.91:57230) 2023-11-24T06:07:46.530 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54ead9d700 1 -- 172.21.15.91:0/2639338971 learned_addr learned my addr 172.21.15.91:0/2639338971 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:46.531 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54eb59e700 1 --2- 172.21.15.91:0/2639338971 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f54ec15bde0 0x7f54ec14ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:46.531 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54ea59c700 1 --2- 172.21.15.91:0/2639338971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f54ec0cf2b0 0x7f54ec0b6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:46.531 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54ead9d700 1 -- 172.21.15.91:0/2639338971 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f54ec15bde0 msgr2=0x7f54ec14ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:46.531 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54ead9d700 1 --2- 172.21.15.91:0/2639338971 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f54ec15bde0 0x7f54ec14ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:46.531 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54ead9d700 1 -- 172.21.15.91:0/2639338971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f54ec0cf2b0 msgr2=0x7f54ec0b6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:46.531 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54ead9d700 1 --2- 172.21.15.91:0/2639338971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f54ec0cf2b0 0x7f54ec0b6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:46.531 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54ead9d700 1 -- 172.21.15.91:0/2639338971 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d4009550 con 0x7f54ec0ce5c0 2023-11-24T06:07:46.531 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54eb59e700 1 --2- 172.21.15.91:0/2639338971 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f54ec15bde0 0x7f54ec14ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:07:46.531 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54ead9d700 1 --2- 172.21.15.91:0/2639338971 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f54ec0ce5c0 0x7f54ec0b4810 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f54e400cbe0 tx=0x7f54e400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:46.531 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54e37fe700 1 -- 172.21.15.91:0/2639338971 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e400ab50 con 0x7f54ec0ce5c0 2023-11-24T06:07:46.532 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54e37fe700 1 -- 172.21.15.91:0/2639338971 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e400acb0 con 0x7f54ec0ce5c0 2023-11-24T06:07:46.532 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54e37fe700 1 -- 172.21.15.91:0/2639338971 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e4007310 con 0x7f54ec0ce5c0 2023-11-24T06:07:46.532 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54f1092700 1 -- 172.21.15.91:0/2639338971 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54ec15c540 con 0x7f54ec0ce5c0 2023-11-24T06:07:46.532 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.525+0000 7f54f1092700 1 -- 172.21.15.91:0/2639338971 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54ec15cb00 con 0x7f54ec0ce5c0 2023-11-24T06:07:46.532 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.526+0000 7f54f1092700 1 -- 172.21.15.91:0/2639338971 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54ec075480 con 0x7f54ec0ce5c0 2023-11-24T06:07:46.532 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.527+0000 7f54e37fe700 1 -- 172.21.15.91:0/2639338971 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f54e4007470 con 0x7f54ec0ce5c0 2023-11-24T06:07:46.532 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.527+0000 7f54e37fe700 1 --2- 172.21.15.91:0/2639338971 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f54cc06ae40 0x7f54cc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:46.532 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.527+0000 7f54e37fe700 1 -- 172.21.15.91:0/2639338971 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f54e4088150 con 0x7f54ec0ce5c0 2023-11-24T06:07:46.532 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.527+0000 7f54ea59c700 1 --2- 172.21.15.91:0/2639338971 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f54cc06ae40 0x7f54cc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:46.533 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.527+0000 7f54ea59c700 1 --2- 172.21.15.91:0/2639338971 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f54cc06ae40 0x7f54cc06d300 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f54d40098f0 tx=0x7f54d4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:46.534 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.532+0000 7f54e37fe700 1 -- 172.21.15.91:0/2639338971 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f54e4054070 con 0x7f54ec0ce5c0 2023-11-24T06:07:46.727 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.724+0000 7f54f1092700 1 -- 172.21.15.91:0/2639338971 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54ec15ccf0 con 0x7f54ec0ce5c0 2023-11-24T06:07:46.728 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.725+0000 7f54e37fe700 1 -- 172.21.15.91:0/2639338971 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f54e4018090 con 0x7f54ec0ce5c0 2023-11-24T06:07:46.728 INFO:teuthology.misc.health.smithi091.stdout: 2023-11-24T06:07:46.728 INFO:teuthology.misc.health.smithi091.stdout:{"epoch":12,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:07:44.391596+0000","last_up_change":"0.000000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:07:46.730 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.727+0000 7f54f1092700 1 -- 172.21.15.91:0/2639338971 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f54cc06ae40 msgr2=0x7f54cc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:46.730 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.727+0000 7f54f1092700 1 --2- 172.21.15.91:0/2639338971 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f54cc06ae40 0x7f54cc06d300 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f54d40098f0 tx=0x7f54d4008030 comp rx=0 tx=0).stop 2023-11-24T06:07:46.730 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.727+0000 7f54f1092700 1 -- 172.21.15.91:0/2639338971 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f54ec0ce5c0 msgr2=0x7f54ec0b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:46.731 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.727+0000 7f54f1092700 1 --2- 172.21.15.91:0/2639338971 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f54ec0ce5c0 0x7f54ec0b4810 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f54e400cbe0 tx=0x7f54e400cef0 comp rx=0 tx=0).stop 2023-11-24T06:07:46.731 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.727+0000 7f54f1092700 1 -- 172.21.15.91:0/2639338971 shutdown_connections 2023-11-24T06:07:46.731 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.727+0000 7f54f1092700 1 --2- 172.21.15.91:0/2639338971 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f54cc06ae40 0x7f54cc06d300 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:46.731 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.727+0000 7f54f1092700 1 --2- 172.21.15.91:0/2639338971 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f54ec15bde0 0x7f54ec14ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:46.731 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.727+0000 7f54f1092700 1 --2- 172.21.15.91:0/2639338971 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f54ec0ce5c0 0x7f54ec0b4810 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:46.731 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.727+0000 7f54f1092700 1 --2- 172.21.15.91:0/2639338971 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f54ec0cf2b0 0x7f54ec0b6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:46.731 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.727+0000 7f54f1092700 1 -- 172.21.15.91:0/2639338971 >> 172.21.15.91:0/2639338971 conn(0x7f54ec0b3720 msgr2=0x7f54ec0b3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:46.731 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.727+0000 7f54f1092700 1 -- 172.21.15.91:0/2639338971 shutdown_connections 2023-11-24T06:07:46.731 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:46.727+0000 7f54f1092700 1 -- 172.21.15.91:0/2639338971 wait complete. 2023-11-24T06:07:46.741 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-11-24T06:07:53.547 DEBUG:teuthology.orchestra.run.smithi091:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-24T06:07:53.670 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.667+0000 7feece2e5700 1 Processor -- start 2023-11-24T06:07:53.670 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.667+0000 7feece2e5700 1 -- start start 2023-11-24T06:07:53.672 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.667+0000 7feece2e5700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7feec806b620 0x7feec80b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:53.672 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.667+0000 7feece2e5700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7feec80b5ab0 0x7feec80b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:53.672 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.668+0000 7feece2e5700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec8071f50 con 0x7feec81478c0 2023-11-24T06:07:53.672 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.668+0000 7feece2e5700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec8072230 con 0x7feec806b620 2023-11-24T06:07:53.672 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.668+0000 7feece2e5700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec8072510 con 0x7feec80b5ab0 2023-11-24T06:07:53.673 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.668+0000 7feec7fff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7feec806b620 0x7feec80b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:53.673 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.668+0000 7feec7fff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7feec806b620 0x7feec80b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57450/0 (socket says 172.21.15.91:57450) 2023-11-24T06:07:53.673 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.668+0000 7feec7fff700 1 -- 172.21.15.91:0/601184629 learned_addr learned my addr 172.21.15.91:0/601184629 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:53.673 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.668+0000 7feec77fe700 1 --2- 172.21.15.91:0/601184629 >> v2:172.21.15.94:3300/0 conn(0x7feec80b5ab0 0x7feec80b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:53.673 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.668+0000 7feec7fff700 1 -- 172.21.15.91:0/601184629 >> v2:172.21.15.94:3300/0 conn(0x7feec80b5ab0 msgr2=0x7feec80b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:53.673 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.668+0000 7feec7fff700 1 --2- 172.21.15.91:0/601184629 >> v2:172.21.15.94:3300/0 conn(0x7feec80b5ab0 0x7feec80b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:53.673 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.668+0000 7feec7fff700 1 -- 172.21.15.91:0/601184629 >> v1:172.21.15.91:6789/0 conn(0x7feec81478c0 legacy=0x7feec8161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:53.673 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.668+0000 7feec7fff700 1 -- 172.21.15.91:0/601184629 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec80727f0 con 0x7feec806b620 2023-11-24T06:07:53.673 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.668+0000 7feec6ffd700 1 -- 172.21.15.91:0/601184629 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268801474 0 0) 0x7feec8071f50 con 0x7feec81478c0 2023-11-24T06:07:53.673 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.668+0000 7feec7fff700 1 --2- 172.21.15.91:0/601184629 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feec806b620 0x7feec80b3a40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7feec806d6f0 tx=0x7feeb8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d74b4779c17adaf server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:53.673 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.668+0000 7feec6ffd700 1 -- 172.21.15.91:0/601184629 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb8011830 con 0x7feec806b620 2023-11-24T06:07:53.673 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.668+0000 7feec6ffd700 1 -- 172.21.15.91:0/601184629 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb8011990 con 0x7feec806b620 2023-11-24T06:07:53.674 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.668+0000 7feec6ffd700 1 -- 172.21.15.91:0/601184629 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb8011d20 con 0x7feec806b620 2023-11-24T06:07:53.674 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.669+0000 7feece2e5700 1 -- 172.21.15.91:0/601184629 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feec806b620 msgr2=0x7feec80b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:53.674 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.669+0000 7feece2e5700 1 --2- 172.21.15.91:0/601184629 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feec806b620 0x7feec80b3a40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7feec806d6f0 tx=0x7feeb8009bd0 comp rx=0 tx=0).stop 2023-11-24T06:07:53.674 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.669+0000 7feece2e5700 1 -- 172.21.15.91:0/601184629 shutdown_connections 2023-11-24T06:07:53.674 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.669+0000 7feece2e5700 1 --2- 172.21.15.91:0/601184629 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feec806b620 0x7feec80b3a40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:53.674 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.669+0000 7feece2e5700 1 --2- 172.21.15.91:0/601184629 >> v2:172.21.15.94:3300/0 conn(0x7feec80b5ab0 0x7feec80b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:53.674 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.669+0000 7feece2e5700 1 -- 172.21.15.91:0/601184629 >> 172.21.15.91:0/601184629 conn(0x7feec80b2740 msgr2=0x7feec80b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:53.674 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.669+0000 7feece2e5700 1 -- 172.21.15.91:0/601184629 shutdown_connections 2023-11-24T06:07:53.674 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.669+0000 7feece2e5700 1 -- 172.21.15.91:0/601184629 wait complete. 2023-11-24T06:07:53.677 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.670+0000 7feece2e5700 1 Processor -- start 2023-11-24T06:07:53.677 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.670+0000 7feece2e5700 1 -- start start 2023-11-24T06:07:53.677 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.671+0000 7feece2e5700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feec806b620 0x7feec80c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:53.677 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.671+0000 7feece2e5700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feec80b5ab0 0x7feec80cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:53.677 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.671+0000 7feece2e5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feec81478c0 0x7feec81e8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:53.677 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.671+0000 7feece2e5700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec80c8160 con 0x7feec806b620 2023-11-24T06:07:53.677 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.671+0000 7feece2e5700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec80c8440 con 0x7feec80b5ab0 2023-11-24T06:07:53.677 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.671+0000 7feece2e5700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec80c8720 con 0x7feec81478c0 2023-11-24T06:07:53.677 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.671+0000 7feec7fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feec806b620 0x7feec80c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:53.677 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.671+0000 7feec7fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feec806b620 0x7feec80c7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57456/0 (socket says 172.21.15.91:57456) 2023-11-24T06:07:53.677 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.671+0000 7feec7fff700 1 -- 172.21.15.91:0/3829040380 learned_addr learned my addr 172.21.15.91:0/3829040380 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:53.678 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.671+0000 7feec7fff700 1 -- 172.21.15.91:0/3829040380 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feec81478c0 msgr2=0x7feec81e8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:53.678 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.671+0000 7feec77fe700 1 --2- 172.21.15.91:0/3829040380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feec80b5ab0 0x7feec80cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:53.678 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.671+0000 7feec7fff700 1 --2- 172.21.15.91:0/3829040380 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feec81478c0 0x7feec81e8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:53.678 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.671+0000 7feec7fff700 1 -- 172.21.15.91:0/3829040380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feec80b5ab0 msgr2=0x7feec80cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:53.678 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.671+0000 7feec7fff700 1 --2- 172.21.15.91:0/3829040380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feec80b5ab0 0x7feec80cd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:53.678 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.671+0000 7feec7fff700 1 -- 172.21.15.91:0/3829040380 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb8009550 con 0x7feec806b620 2023-11-24T06:07:53.678 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.672+0000 7feec7fff700 1 --2- 172.21.15.91:0/3829040380 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feec806b620 0x7feec80c7b80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feec80d1bd0 tx=0x7feeb8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:53.678 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.672+0000 7feec4ff9700 1 -- 172.21.15.91:0/3829040380 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb80036a0 con 0x7feec806b620 2023-11-24T06:07:53.678 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.672+0000 7feec4ff9700 1 -- 172.21.15.91:0/3829040380 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb8003800 con 0x7feec806b620 2023-11-24T06:07:53.678 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.672+0000 7feec4ff9700 1 -- 172.21.15.91:0/3829040380 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb8003b90 con 0x7feec806b620 2023-11-24T06:07:53.678 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.672+0000 7feece2e5700 1 -- 172.21.15.91:0/3829040380 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec80c8a00 con 0x7feec806b620 2023-11-24T06:07:53.679 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.672+0000 7feece2e5700 1 -- 172.21.15.91:0/3829040380 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec8146b00 con 0x7feec806b620 2023-11-24T06:07:53.679 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.673+0000 7feece2e5700 1 -- 172.21.15.91:0/3829040380 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec80c8f20 con 0x7feec806b620 2023-11-24T06:07:53.679 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.673+0000 7feec4ff9700 1 -- 172.21.15.91:0/3829040380 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7feeb8003cf0 con 0x7feec806b620 2023-11-24T06:07:53.679 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.674+0000 7feec4ff9700 1 --2- 172.21.15.91:0/3829040380 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7feeb406aed0 0x7feeb406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:53.679 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.674+0000 7feec4ff9700 1 -- 172.21.15.91:0/3829040380 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7feeb8021030 con 0x7feec806b620 2023-11-24T06:07:53.679 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.674+0000 7feec77fe700 1 --2- 172.21.15.91:0/3829040380 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7feeb406aed0 0x7feeb406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:53.679 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.674+0000 7feec77fe700 1 --2- 172.21.15.91:0/3829040380 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7feeb406aed0 0x7feeb406d390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7feec81e9a10 tx=0x7feebc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:53.681 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.678+0000 7feec4ff9700 1 -- 172.21.15.91:0/3829040380 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feec80c8f20 con 0x7feec806b620 2023-11-24T06:07:53.873 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.870+0000 7feece2e5700 1 -- 172.21.15.91:0/3829040380 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feec80c8f20 con 0x7feec806b620 2023-11-24T06:07:53.874 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.871+0000 7feec4ff9700 1 -- 172.21.15.91:0/3829040380 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12766 (secure 0 0 0) 0x7feeb802b090 con 0x7feec806b620 2023-11-24T06:07:53.875 INFO:teuthology.misc.health.smithi091.stdout: 2023-11-24T06:07:53.875 INFO:teuthology.misc.health.smithi091.stdout:{"epoch":16,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:07:49.871796+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:07:53.877 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.873+0000 7feece2e5700 1 -- 172.21.15.91:0/3829040380 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7feeb406aed0 msgr2=0x7feeb406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:53.877 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.873+0000 7feece2e5700 1 --2- 172.21.15.91:0/3829040380 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7feeb406aed0 0x7feeb406d390 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7feec81e9a10 tx=0x7feebc008040 comp rx=0 tx=0).stop 2023-11-24T06:07:53.877 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.873+0000 7feece2e5700 1 -- 172.21.15.91:0/3829040380 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feec806b620 msgr2=0x7feec80c7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:53.878 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.873+0000 7feece2e5700 1 --2- 172.21.15.91:0/3829040380 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feec806b620 0x7feec80c7b80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feec80d1bd0 tx=0x7feeb8004750 comp rx=0 tx=0).stop 2023-11-24T06:07:53.878 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.874+0000 7feece2e5700 1 -- 172.21.15.91:0/3829040380 shutdown_connections 2023-11-24T06:07:53.878 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.874+0000 7feece2e5700 1 --2- 172.21.15.91:0/3829040380 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7feeb406aed0 0x7feeb406d390 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:53.878 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.874+0000 7feece2e5700 1 --2- 172.21.15.91:0/3829040380 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feec81478c0 0x7feec81e8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:53.878 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.874+0000 7feece2e5700 1 --2- 172.21.15.91:0/3829040380 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feec806b620 0x7feec80c7b80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:53.878 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.874+0000 7feece2e5700 1 --2- 172.21.15.91:0/3829040380 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feec80b5ab0 0x7feec80cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:53.878 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.874+0000 7feece2e5700 1 -- 172.21.15.91:0/3829040380 >> 172.21.15.91:0/3829040380 conn(0x7feec80b2740 msgr2=0x7feec80b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:53.879 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.874+0000 7feece2e5700 1 -- 172.21.15.91:0/3829040380 shutdown_connections 2023-11-24T06:07:53.879 INFO:teuthology.misc.health.smithi091.stderr:2023-11-24T06:07:53.874+0000 7feece2e5700 1 -- 172.21.15.91:0/3829040380 wait complete. 2023-11-24T06:07:53.888 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-11-24T06:07:53.888 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-24T06:07:53.888 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T06:07:53.889 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T06:07:53.890 INFO:tasks.ceph.mds.a:Started 2023-11-24T06:07:53.891 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T06:07:53.891 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T06:07:53.892 INFO:tasks.ceph.mds.c:Started 2023-11-24T06:07:53.893 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T06:07:53.893 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T06:07:53.895 INFO:tasks.ceph.mds.b:Started 2023-11-24T06:07:53.895 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T06:07:53.895 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T06:07:53.897 INFO:tasks.ceph.mds.d:Started 2023-11-24T06:07:53.897 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-11-24T06:07:53.897 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:07:53.969 INFO:tasks.ceph.mds.c.smithi091.stdout:starting mds.c at 2023-11-24T06:07:53.970 INFO:tasks.ceph.mds.a.smithi091.stdout:starting mds.a at 2023-11-24T06:07:53.984 INFO:tasks.ceph.mds.b.smithi094.stdout:starting mds.b at 2023-11-24T06:07:53.984 INFO:tasks.ceph.mds.d.smithi094.stdout:starting mds.d at 2023-11-24T06:07:54.026 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.023+0000 7f6c393c7700 1 Processor -- start 2023-11-24T06:07:54.026 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.023+0000 7f6c393c7700 1 -- start start 2023-11-24T06:07:54.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.024+0000 7f6c393c7700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6c3406b620 0x7f6c340b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.024+0000 7f6c393c7700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6c3406ee60 0x7f6c340b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.024+0000 7f6c393c7700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c34071f50 con 0x7f6c340b5ab0 2023-11-24T06:07:54.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.024+0000 7f6c393c7700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c34072230 con 0x7f6c3406ee60 2023-11-24T06:07:54.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.024+0000 7f6c393c7700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c34072510 con 0x7f6c3406b620 2023-11-24T06:07:54.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.024+0000 7f6c337fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6c3406ee60 0x7f6c340b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.024+0000 7f6c32ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6c3406b620 0x7f6c340b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.024+0000 7f6c327fc700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f6c340b5ab0 0x7f6c34161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.91:54216/0 (socket says 172.21.15.91:54216) 2023-11-24T06:07:54.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.024+0000 7f6c32ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6c3406b620 0x7f6c340b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:40920/0 (socket says 172.21.15.91:40920) 2023-11-24T06:07:54.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.024+0000 7f6c337fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6c3406ee60 0x7f6c340b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:41884/0 (socket says 172.21.15.91:41884) 2023-11-24T06:07:54.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.025+0000 7f6c327fc700 1 -- 172.21.15.91:0/3588226502 learned_addr learned my addr 172.21.15.91:0/3588226502 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:07:54.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.025+0000 7f6c337fe700 1 -- 172.21.15.91:0/3588226502 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6c3406b620 msgr2=0x7f6c340b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:54.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.025+0000 7f6c337fe700 1 --2- 172.21.15.91:0/3588226502 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6c3406b620 0x7f6c340b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.025+0000 7f6c337fe700 1 -- 172.21.15.91:0/3588226502 >> v1:172.21.15.94:6789/0 conn(0x7f6c340b5ab0 legacy=0x7f6c34161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.025+0000 7f6c337fe700 1 -- 172.21.15.91:0/3588226502 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c340727f0 con 0x7f6c3406ee60 2023-11-24T06:07:54.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.025+0000 7f6c31ffb700 1 -- 172.21.15.91:0/3588226502 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869448446 0 0) 0x7f6c34071f50 con 0x7f6c340b5ab0 2023-11-24T06:07:54.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.025+0000 7f6c32ffd700 1 --2- 172.21.15.91:0/3588226502 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6c3406b620 0x7f6c340b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:07:54.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.025+0000 7f6c337fe700 1 --2- 172.21.15.91:0/3588226502 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c3406ee60 0x7f6c340b3f80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009840 tx=0x7f6c24009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc48386081bd0f6d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:54.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.025+0000 7f6c31ffb700 1 -- 172.21.15.91:0/3588226502 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c24011830 con 0x7f6c3406ee60 2023-11-24T06:07:54.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.025+0000 7f6c31ffb700 1 -- 172.21.15.91:0/3588226502 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c24011990 con 0x7f6c3406ee60 2023-11-24T06:07:54.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.025+0000 7f6c31ffb700 1 -- 172.21.15.91:0/3588226502 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c24011d20 con 0x7f6c3406ee60 2023-11-24T06:07:54.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.026+0000 7f6c393c7700 1 -- 172.21.15.91:0/3588226502 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c3406ee60 msgr2=0x7f6c340b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.031 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.026+0000 7f6c393c7700 1 --2- 172.21.15.91:0/3588226502 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c3406ee60 0x7f6c340b3f80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009840 tx=0x7f6c24009b50 comp rx=0 tx=0).stop 2023-11-24T06:07:54.031 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.026+0000 7f6c393c7700 1 -- 172.21.15.91:0/3588226502 shutdown_connections 2023-11-24T06:07:54.031 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.026+0000 7f6c393c7700 1 --2- 172.21.15.91:0/3588226502 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c3406ee60 0x7f6c340b3f80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.031 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.026+0000 7f6c393c7700 1 --2- 172.21.15.91:0/3588226502 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6c3406b620 0x7f6c340b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.031 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.026+0000 7f6c393c7700 1 -- 172.21.15.91:0/3588226502 >> 172.21.15.91:0/3588226502 conn(0x7f6c340b2740 msgr2=0x7f6c340b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:54.031 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.026+0000 7f6c393c7700 1 -- 172.21.15.91:0/3588226502 shutdown_connections 2023-11-24T06:07:54.031 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.026+0000 7f6c393c7700 1 -- 172.21.15.91:0/3588226502 wait complete. 2023-11-24T06:07:54.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.027+0000 7f6c393c7700 1 Processor -- start 2023-11-24T06:07:54.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.027+0000 7f6c393c7700 1 -- start start 2023-11-24T06:07:54.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.028+0000 7f6c393c7700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6c3406b620 0x7f6c341e8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.028+0000 7f6c393c7700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6c3406ee60 0x7f6c340c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.028+0000 7f6c393c7700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c340b5ab0 0x7f6c341430d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.028+0000 7f6c393c7700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c340c6320 con 0x7f6c3406ee60 2023-11-24T06:07:54.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.028+0000 7f6c393c7700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c340c6600 con 0x7f6c340b5ab0 2023-11-24T06:07:54.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.028+0000 7f6c393c7700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c340c68e0 con 0x7f6c3406b620 2023-11-24T06:07:54.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.028+0000 7f6c327fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6c3406ee60 0x7f6c340c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.028+0000 7f6c327fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6c3406ee60 0x7f6c340c59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57496/0 (socket says 172.21.15.91:57496) 2023-11-24T06:07:54.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.028+0000 7f6c327fc700 1 -- 172.21.15.91:0/3325598655 learned_addr learned my addr 172.21.15.91:0/3325598655 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:54.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.029+0000 7f6c337fe700 1 --2- 172.21.15.91:0/3325598655 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c340b5ab0 0x7f6c341430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.029+0000 7f6c32ffd700 1 --2- 172.21.15.91:0/3325598655 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6c3406b620 0x7f6c341e8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.029+0000 7f6c327fc700 1 -- 172.21.15.91:0/3325598655 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6c3406b620 msgr2=0x7f6c341e8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.029+0000 7f6c327fc700 1 --2- 172.21.15.91:0/3325598655 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6c3406b620 0x7f6c341e8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.029+0000 7f6c327fc700 1 -- 172.21.15.91:0/3325598655 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c340b5ab0 msgr2=0x7f6c341430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.035 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.029+0000 7f6c327fc700 1 --2- 172.21.15.91:0/3325598655 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c340b5ab0 0x7f6c341430d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.035 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.029+0000 7f6c327fc700 1 -- 172.21.15.91:0/3325598655 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2c009a70 con 0x7f6c3406ee60 2023-11-24T06:07:54.035 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.029+0000 7f6c327fc700 1 --2- 172.21.15.91:0/3325598655 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6c3406ee60 0x7f6c340c59b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6c2800c520 tx=0x7f6c2800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:54.035 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.029+0000 7f6c1b7fe700 1 -- 172.21.15.91:0/3325598655 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c28004c80 con 0x7f6c3406ee60 2023-11-24T06:07:54.035 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.029+0000 7f6c1b7fe700 1 -- 172.21.15.91:0/3325598655 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c28004de0 con 0x7f6c3406ee60 2023-11-24T06:07:54.035 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.029+0000 7f6c393c7700 1 -- 172.21.15.91:0/3325598655 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c240094a0 con 0x7f6c3406ee60 2023-11-24T06:07:54.035 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.029+0000 7f6c1b7fe700 1 -- 172.21.15.91:0/3325598655 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c280072d0 con 0x7f6c3406ee60 2023-11-24T06:07:54.035 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.029+0000 7f6c393c7700 1 -- 172.21.15.91:0/3325598655 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c341e9240 con 0x7f6c3406ee60 2023-11-24T06:07:54.035 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.030+0000 7f6c393c7700 1 -- 172.21.15.91:0/3325598655 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c340c6e00 con 0x7f6c3406ee60 2023-11-24T06:07:54.035 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.031+0000 7f6c1b7fe700 1 -- 172.21.15.91:0/3325598655 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6c28007430 con 0x7f6c3406ee60 2023-11-24T06:07:54.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.031+0000 7f6c1b7fe700 1 --2- 172.21.15.91:0/3325598655 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6c2006aed0 0x7f6c2006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.031+0000 7f6c1b7fe700 1 -- 172.21.15.91:0/3325598655 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6c28088df0 con 0x7f6c3406ee60 2023-11-24T06:07:54.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.031+0000 7f6c32ffd700 1 --2- 172.21.15.91:0/3325598655 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6c2006aed0 0x7f6c2006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.031+0000 7f6c32ffd700 1 --2- 172.21.15.91:0/3325598655 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6c2006aed0 0x7f6c2006d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009fd0 tx=0x7f6c2c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:54.038 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.035+0000 7f6c1b7fe700 1 -- 172.21.15.91:0/3325598655 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6c340c6e00 con 0x7f6c3406ee60 2023-11-24T06:07:54.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.227+0000 7f6c393c7700 1 -- 172.21.15.91:0/3325598655 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c340c6e00 con 0x7f6c3406ee60 2023-11-24T06:07:54.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.228+0000 7f6c1b7fe700 1 -- 172.21.15.91:0/3325598655 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12766 (secure 0 0 0) 0x7f6c340c6e00 con 0x7f6c3406ee60 2023-11-24T06:07:54.231 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:54.231 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":16,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:07:49.871796+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:07:54.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.230+0000 7f6c393c7700 1 -- 172.21.15.91:0/3325598655 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6c2006aed0 msgr2=0x7f6c2006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.230+0000 7f6c393c7700 1 --2- 172.21.15.91:0/3325598655 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6c2006aed0 0x7f6c2006d390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c009fd0 tx=0x7f6c2c00b040 comp rx=0 tx=0).stop 2023-11-24T06:07:54.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.230+0000 7f6c393c7700 1 -- 172.21.15.91:0/3325598655 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6c3406ee60 msgr2=0x7f6c340c59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.230+0000 7f6c393c7700 1 --2- 172.21.15.91:0/3325598655 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6c3406ee60 0x7f6c340c59b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6c2800c520 tx=0x7f6c2800c830 comp rx=0 tx=0).stop 2023-11-24T06:07:54.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.230+0000 7f6c393c7700 1 -- 172.21.15.91:0/3325598655 shutdown_connections 2023-11-24T06:07:54.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.230+0000 7f6c393c7700 1 --2- 172.21.15.91:0/3325598655 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6c2006aed0 0x7f6c2006d390 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.230+0000 7f6c393c7700 1 --2- 172.21.15.91:0/3325598655 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6c340b5ab0 0x7f6c341430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.230+0000 7f6c393c7700 1 --2- 172.21.15.91:0/3325598655 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6c3406b620 0x7f6c341e8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.230+0000 7f6c393c7700 1 --2- 172.21.15.91:0/3325598655 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6c3406ee60 0x7f6c340c59b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.230+0000 7f6c393c7700 1 -- 172.21.15.91:0/3325598655 >> 172.21.15.91:0/3325598655 conn(0x7f6c340b2740 msgr2=0x7f6c34146e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:54.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.230+0000 7f6c393c7700 1 -- 172.21.15.91:0/3325598655 shutdown_connections 2023-11-24T06:07:54.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.230+0000 7f6c393c7700 1 -- 172.21.15.91:0/3325598655 wait complete. 2023-11-24T06:07:54.245 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:07:54.245 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:07:54.369 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.365+0000 7fc06d0f5700 1 Processor -- start 2023-11-24T06:07:54.369 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.366+0000 7fc06d0f5700 1 -- start start 2023-11-24T06:07:54.371 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.366+0000 7fc06d0f5700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fc06815e1c0 0x7fc06806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.371 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.366+0000 7fc06d0f5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc06806fc50 0x7fc06806ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.371 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.366+0000 7fc06d0f5700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0680d1230 con 0x7fc0680b0e90 2023-11-24T06:07:54.371 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.366+0000 7fc06d0f5700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0680d0c70 con 0x7fc06815e1c0 2023-11-24T06:07:54.371 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.366+0000 7fc06d0f5700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0680d0f50 con 0x7fc06806fc50 2023-11-24T06:07:54.371 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.366+0000 7fc066d9d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fc06815e1c0 0x7fc06806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.366+0000 7fc06659c700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fc0680b0e90 0x7fc06806f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52436/0 (socket says 172.21.15.91:52436) 2023-11-24T06:07:54.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.366+0000 7fc06659c700 1 -- 172.21.15.91:0/932615435 learned_addr learned my addr 172.21.15.91:0/932615435 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:07:54.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.367+0000 7fc06759e700 1 --2- 172.21.15.91:0/932615435 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc06806fc50 0x7fc06806ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.367+0000 7fc066d9d700 1 -- 172.21.15.91:0/932615435 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc06806fc50 msgr2=0x7fc06806ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:54.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.367+0000 7fc066d9d700 1 --2- 172.21.15.91:0/932615435 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc06806fc50 0x7fc06806ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.367+0000 7fc066d9d700 1 -- 172.21.15.91:0/932615435 >> v1:172.21.15.91:6789/0 conn(0x7fc0680b0e90 legacy=0x7fc06806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.367+0000 7fc066d9d700 1 -- 172.21.15.91:0/932615435 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06806c210 con 0x7fc06815e1c0 2023-11-24T06:07:54.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.367+0000 7fc065d9b700 1 -- 172.21.15.91:0/932615435 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577278560 0 0) 0x7fc0680d1230 con 0x7fc0680b0e90 2023-11-24T06:07:54.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.367+0000 7fc066d9d700 1 --2- 172.21.15.91:0/932615435 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc06815e1c0 0x7fc06806eef0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc05c009840 tx=0x7fc05c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11473e6c7389e43d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:54.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.367+0000 7fc065d9b700 1 -- 172.21.15.91:0/932615435 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c011830 con 0x7fc06815e1c0 2023-11-24T06:07:54.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.367+0000 7fc065d9b700 1 -- 172.21.15.91:0/932615435 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05c011990 con 0x7fc06815e1c0 2023-11-24T06:07:54.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.367+0000 7fc065d9b700 1 -- 172.21.15.91:0/932615435 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c011d20 con 0x7fc06815e1c0 2023-11-24T06:07:54.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.368+0000 7fc06d0f5700 1 -- 172.21.15.91:0/932615435 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc06815e1c0 msgr2=0x7fc06806eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.368+0000 7fc06d0f5700 1 --2- 172.21.15.91:0/932615435 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc06815e1c0 0x7fc06806eef0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc05c009840 tx=0x7fc05c009b50 comp rx=0 tx=0).stop 2023-11-24T06:07:54.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.368+0000 7fc06d0f5700 1 -- 172.21.15.91:0/932615435 shutdown_connections 2023-11-24T06:07:54.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.368+0000 7fc06d0f5700 1 --2- 172.21.15.91:0/932615435 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc06806fc50 0x7fc06806ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.368+0000 7fc06d0f5700 1 --2- 172.21.15.91:0/932615435 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc06815e1c0 0x7fc06806eef0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.368+0000 7fc06d0f5700 1 -- 172.21.15.91:0/932615435 >> 172.21.15.91:0/932615435 conn(0x7fc0680b3720 msgr2=0x7fc0680b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:54.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.368+0000 7fc06d0f5700 1 -- 172.21.15.91:0/932615435 shutdown_connections 2023-11-24T06:07:54.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.368+0000 7fc06d0f5700 1 -- 172.21.15.91:0/932615435 wait complete. 2023-11-24T06:07:54.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.369+0000 7fc06d0f5700 1 Processor -- start 2023-11-24T06:07:54.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.369+0000 7fc06d0f5700 1 -- start start 2023-11-24T06:07:54.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc06d0f5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc06806fc50 0x7fc0680b2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc06d0f5700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc0680b0e90 0x7fc0680b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc06d0f5700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc06815e1c0 0x7fc0681f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc06d0f5700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06814a660 con 0x7fc0680b0e90 2023-11-24T06:07:54.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc06d0f5700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06815efa0 con 0x7fc06815e1c0 2023-11-24T06:07:54.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc06d0f5700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0681504f0 con 0x7fc06806fc50 2023-11-24T06:07:54.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc06659c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc0680b0e90 0x7fc0680b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc066d9d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc06806fc50 0x7fc0680b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc06659c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc0680b0e90 0x7fc0680b2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57508/0 (socket says 172.21.15.91:57508) 2023-11-24T06:07:54.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc06659c700 1 -- 172.21.15.91:0/3679936869 learned_addr learned my addr 172.21.15.91:0/3679936869 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:54.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc06759e700 1 --2- 172.21.15.91:0/3679936869 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc06815e1c0 0x7fc0681f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc06659c700 1 -- 172.21.15.91:0/3679936869 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc06806fc50 msgr2=0x7fc0680b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc06659c700 1 --2- 172.21.15.91:0/3679936869 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc06806fc50 0x7fc0680b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc06659c700 1 -- 172.21.15.91:0/3679936869 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc06815e1c0 msgr2=0x7fc0681f2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc06659c700 1 --2- 172.21.15.91:0/3679936869 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc06815e1c0 0x7fc0681f2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc06659c700 1 -- 172.21.15.91:0/3679936869 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05c0094a0 con 0x7fc0680b0e90 2023-11-24T06:07:54.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.370+0000 7fc06659c700 1 --2- 172.21.15.91:0/3679936869 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc0680b0e90 0x7fc0680b2d60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc0580011e0 tx=0x7fc05800ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:54.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.371+0000 7fc04f7fe700 1 -- 172.21.15.91:0/3679936869 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc058004c80 con 0x7fc0680b0e90 2023-11-24T06:07:54.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.371+0000 7fc04f7fe700 1 -- 172.21.15.91:0/3679936869 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc058004e00 con 0x7fc0680b0e90 2023-11-24T06:07:54.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.371+0000 7fc04f7fe700 1 -- 172.21.15.91:0/3679936869 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc058007310 con 0x7fc0680b0e90 2023-11-24T06:07:54.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.371+0000 7fc06d0f5700 1 -- 172.21.15.91:0/3679936869 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc06815e720 con 0x7fc0680b0e90 2023-11-24T06:07:54.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.371+0000 7fc06d0f5700 1 -- 172.21.15.91:0/3679936869 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06815ecc0 con 0x7fc0680b0e90 2023-11-24T06:07:54.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.372+0000 7fc06d0f5700 1 -- 172.21.15.91:0/3679936869 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc06814a7d0 con 0x7fc0680b0e90 2023-11-24T06:07:54.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.372+0000 7fc04f7fe700 1 -- 172.21.15.91:0/3679936869 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc058007470 con 0x7fc0680b0e90 2023-11-24T06:07:54.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.372+0000 7fc04f7fe700 1 --2- 172.21.15.91:0/3679936869 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc05406aed0 0x7fc05406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.372+0000 7fc04f7fe700 1 -- 172.21.15.91:0/3679936869 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc058088eb0 con 0x7fc0680b0e90 2023-11-24T06:07:54.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.372+0000 7fc066d9d700 1 --2- 172.21.15.91:0/3679936869 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc05406aed0 0x7fc05406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.373+0000 7fc066d9d700 1 --2- 172.21.15.91:0/3679936869 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc05406aed0 0x7fc05406d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc05c0090b0 tx=0x7fc05c01c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:54.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.377+0000 7fc04f7fe700 1 -- 172.21.15.91:0/3679936869 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc058054340 con 0x7fc0680b0e90 2023-11-24T06:07:54.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.564+0000 7fc06d0f5700 1 -- 172.21.15.91:0/3679936869 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc068075480 con 0x7fc0680b0e90 2023-11-24T06:07:54.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.564+0000 7fc04f7fe700 1 -- 172.21.15.91:0/3679936869 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc058057960 con 0x7fc0680b0e90 2023-11-24T06:07:54.568 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:07:54.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.566+0000 7fc06d0f5700 1 -- 172.21.15.91:0/3679936869 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc05406aed0 msgr2=0x7fc05406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.566+0000 7fc06d0f5700 1 --2- 172.21.15.91:0/3679936869 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc05406aed0 0x7fc05406d390 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc05c0090b0 tx=0x7fc05c01c040 comp rx=0 tx=0).stop 2023-11-24T06:07:54.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.566+0000 7fc06d0f5700 1 -- 172.21.15.91:0/3679936869 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc0680b0e90 msgr2=0x7fc0680b2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.566+0000 7fc06d0f5700 1 --2- 172.21.15.91:0/3679936869 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc0680b0e90 0x7fc0680b2d60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc0580011e0 tx=0x7fc05800ca30 comp rx=0 tx=0).stop 2023-11-24T06:07:54.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.566+0000 7fc06d0f5700 1 -- 172.21.15.91:0/3679936869 shutdown_connections 2023-11-24T06:07:54.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.566+0000 7fc06d0f5700 1 --2- 172.21.15.91:0/3679936869 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc05406aed0 0x7fc05406d390 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.566+0000 7fc06d0f5700 1 --2- 172.21.15.91:0/3679936869 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc06815e1c0 0x7fc0681f2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.566+0000 7fc06d0f5700 1 --2- 172.21.15.91:0/3679936869 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc06806fc50 0x7fc0680b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.566+0000 7fc06d0f5700 1 --2- 172.21.15.91:0/3679936869 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc0680b0e90 0x7fc0680b2d60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.566+0000 7fc06d0f5700 1 -- 172.21.15.91:0/3679936869 >> 172.21.15.91:0/3679936869 conn(0x7fc0680b3720 msgr2=0x7fc0680b3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:54.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.566+0000 7fc06d0f5700 1 -- 172.21.15.91:0/3679936869 shutdown_connections 2023-11-24T06:07:54.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.566+0000 7fc06d0f5700 1 -- 172.21.15.91:0/3679936869 wait complete. 2023-11-24T06:07:54.581 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:07:54.701 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.698+0000 7fad4522d700 1 Processor -- start 2023-11-24T06:07:54.701 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.698+0000 7fad4522d700 1 -- start start 2023-11-24T06:07:54.703 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.699+0000 7fad4522d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fad400cecd0 0x7fad4015c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.703 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.699+0000 7fad4522d700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad400d0a90 con 0x7fad4015bde0 2023-11-24T06:07:54.703 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.699+0000 7fad4522d700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4006ee10 con 0x7fad400cf2b0 2023-11-24T06:07:54.703 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.699+0000 7fad4522d700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad4006f0f0 con 0x7fad400cecd0 2023-11-24T06:07:54.703 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.699+0000 7fad3f59e700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fad400cecd0 0x7fad4015c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.703 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.699+0000 7fad3ed9d700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fad4015bde0 0x7fad400d01b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52448/0 (socket says 172.21.15.91:52448) 2023-11-24T06:07:54.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.699+0000 7fad3ed9d700 1 -- 172.21.15.91:0/2889868064 learned_addr learned my addr 172.21.15.91:0/2889868064 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:07:54.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.699+0000 7fad3f59e700 1 -- 172.21.15.91:0/2889868064 >> v1:172.21.15.94:6789/0 conn(0x7fad400cf2b0 legacy=0x7fad400ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.699+0000 7fad3f59e700 1 -- 172.21.15.91:0/2889868064 >> v1:172.21.15.91:6789/0 conn(0x7fad4015bde0 legacy=0x7fad400d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.699+0000 7fad3f59e700 1 -- 172.21.15.91:0/2889868064 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4006f3d0 con 0x7fad400cecd0 2023-11-24T06:07:54.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.699+0000 7fad3f59e700 1 --2- 172.21.15.91:0/2889868064 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad400cecd0 0x7fad4015c1c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fad38009d90 tx=0x7fad3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cdc78c9498b7d02 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:54.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.699+0000 7fad3dd9b700 1 -- 172.21.15.91:0/2889868064 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3800e070 con 0x7fad400cecd0 2023-11-24T06:07:54.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.699+0000 7fad3dd9b700 1 -- 172.21.15.91:0/2889868064 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3800bbd0 con 0x7fad400cecd0 2023-11-24T06:07:54.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.699+0000 7fad3dd9b700 1 -- 172.21.15.91:0/2889868064 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad38012430 con 0x7fad400cecd0 2023-11-24T06:07:54.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.700+0000 7fad4522d700 1 -- 172.21.15.91:0/2889868064 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad400cecd0 msgr2=0x7fad4015c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.700+0000 7fad4522d700 1 --2- 172.21.15.91:0/2889868064 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad400cecd0 0x7fad4015c1c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fad38009d90 tx=0x7fad3800b330 comp rx=0 tx=0).stop 2023-11-24T06:07:54.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.700+0000 7fad4522d700 1 -- 172.21.15.91:0/2889868064 shutdown_connections 2023-11-24T06:07:54.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.700+0000 7fad4522d700 1 --2- 172.21.15.91:0/2889868064 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad400cecd0 0x7fad4015c1c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.705 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.700+0000 7fad4522d700 1 -- 172.21.15.91:0/2889868064 >> 172.21.15.91:0/2889868064 conn(0x7fad400b3720 msgr2=0x7fad400b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:54.705 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.700+0000 7fad4522d700 1 -- 172.21.15.91:0/2889868064 shutdown_connections 2023-11-24T06:07:54.705 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.700+0000 7fad4522d700 1 -- 172.21.15.91:0/2889868064 wait complete. 2023-11-24T06:07:54.707 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.701+0000 7fad4522d700 1 Processor -- start 2023-11-24T06:07:54.707 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.701+0000 7fad4522d700 1 -- start start 2023-11-24T06:07:54.707 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad4522d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad400cecd0 0x7fad4006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.707 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad4522d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad400cf2b0 0x7fad40152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.707 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad4522d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad4015bde0 0x7fad40152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.707 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad4522d700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad40164440 con 0x7fad4015bde0 2023-11-24T06:07:54.707 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad4522d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad40164a00 con 0x7fad400cecd0 2023-11-24T06:07:54.708 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad4522d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4015e5c0 con 0x7fad400cf2b0 2023-11-24T06:07:54.708 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad3f59e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad4015bde0 0x7fad40152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.708 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad3f59e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad4015bde0 0x7fad40152f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57522/0 (socket says 172.21.15.91:57522) 2023-11-24T06:07:54.708 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad3f59e700 1 -- 172.21.15.91:0/3387517199 learned_addr learned my addr 172.21.15.91:0/3387517199 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:54.708 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad3ed9d700 1 --2- 172.21.15.91:0/3387517199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad400cecd0 0x7fad4006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.708 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad3e59c700 1 --2- 172.21.15.91:0/3387517199 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad400cf2b0 0x7fad40152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.708 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad3f59e700 1 -- 172.21.15.91:0/3387517199 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad400cf2b0 msgr2=0x7fad40152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.708 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad3f59e700 1 --2- 172.21.15.91:0/3387517199 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad400cf2b0 0x7fad40152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.708 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad3f59e700 1 -- 172.21.15.91:0/3387517199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad400cecd0 msgr2=0x7fad4006d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.708 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad3f59e700 1 --2- 172.21.15.91:0/3387517199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad400cecd0 0x7fad4006d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.708 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad3f59e700 1 -- 172.21.15.91:0/3387517199 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad38009a70 con 0x7fad4015bde0 2023-11-24T06:07:54.708 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.702+0000 7fad3e59c700 1 --2- 172.21.15.91:0/3387517199 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad400cf2b0 0x7fad40152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:07:54.709 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.703+0000 7fad3f59e700 1 --2- 172.21.15.91:0/3387517199 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad4015bde0 0x7fad40152f60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fad38014040 tx=0x7fad38004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:54.709 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.703+0000 7fad277fe700 1 -- 172.21.15.91:0/3387517199 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad380098f0 con 0x7fad4015bde0 2023-11-24T06:07:54.709 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.703+0000 7fad277fe700 1 -- 172.21.15.91:0/3387517199 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad38020070 con 0x7fad4015bde0 2023-11-24T06:07:54.709 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.703+0000 7fad4522d700 1 -- 172.21.15.91:0/3387517199 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4015e8a0 con 0x7fad4015bde0 2023-11-24T06:07:54.709 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.703+0000 7fad277fe700 1 -- 172.21.15.91:0/3387517199 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad38004280 con 0x7fad4015bde0 2023-11-24T06:07:54.709 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.703+0000 7fad4522d700 1 -- 172.21.15.91:0/3387517199 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4006df30 con 0x7fad4015bde0 2023-11-24T06:07:54.709 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.704+0000 7fad4522d700 1 -- 172.21.15.91:0/3387517199 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad4006e0f0 con 0x7fad4015bde0 2023-11-24T06:07:54.709 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.704+0000 7fad277fe700 1 -- 172.21.15.91:0/3387517199 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fad380043e0 con 0x7fad4015bde0 2023-11-24T06:07:54.710 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.704+0000 7fad277fe700 1 --2- 172.21.15.91:0/3387517199 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad2c06aed0 0x7fad2c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:54.710 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.705+0000 7fad277fe700 1 -- 172.21.15.91:0/3387517199 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fad38022070 con 0x7fad4015bde0 2023-11-24T06:07:54.710 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.705+0000 7fad3ed9d700 1 --2- 172.21.15.91:0/3387517199 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad2c06aed0 0x7fad2c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:54.710 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.705+0000 7fad3ed9d700 1 --2- 172.21.15.91:0/3387517199 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad2c06aed0 0x7fad2c06d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fad30005910 tx=0x7fad30009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:54.712 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.709+0000 7fad277fe700 1 -- 172.21.15.91:0/3387517199 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad4006e0f0 con 0x7fad4015bde0 2023-11-24T06:07:54.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.947+0000 7fad4522d700 1 -- 172.21.15.91:0/3387517199 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad4006e0c0 con 0x7fad4015bde0 2023-11-24T06:07:54.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.947+0000 7fad277fe700 1 -- 172.21.15.91:0/3387517199 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7fad38056530 con 0x7fad4015bde0 2023-11-24T06:07:54.951 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:54.951 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6837/1644079645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":1644079645},{"type":"v1","addr":"172.21.15.91:6837","nonce":1644079645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-11-24T06:07:54.953 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.949+0000 7fad4522d700 1 -- 172.21.15.91:0/3387517199 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad2c06aed0 msgr2=0x7fad2c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.953 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.949+0000 7fad4522d700 1 --2- 172.21.15.91:0/3387517199 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad2c06aed0 0x7fad2c06d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fad30005910 tx=0x7fad30009040 comp rx=0 tx=0).stop 2023-11-24T06:07:54.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.949+0000 7fad4522d700 1 -- 172.21.15.91:0/3387517199 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad4015bde0 msgr2=0x7fad40152f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:54.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.949+0000 7fad4522d700 1 --2- 172.21.15.91:0/3387517199 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad4015bde0 0x7fad40152f60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fad38014040 tx=0x7fad38004ad0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.949+0000 7fad4522d700 1 -- 172.21.15.91:0/3387517199 shutdown_connections 2023-11-24T06:07:54.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.949+0000 7fad4522d700 1 --2- 172.21.15.91:0/3387517199 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad2c06aed0 0x7fad2c06d390 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.949+0000 7fad4522d700 1 --2- 172.21.15.91:0/3387517199 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad400cecd0 0x7fad4006d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.949+0000 7fad4522d700 1 --2- 172.21.15.91:0/3387517199 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad4015bde0 0x7fad40152f60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.949+0000 7fad4522d700 1 --2- 172.21.15.91:0/3387517199 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad400cf2b0 0x7fad40152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:54.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.949+0000 7fad4522d700 1 -- 172.21.15.91:0/3387517199 >> 172.21.15.91:0/3387517199 conn(0x7fad400b3720 msgr2=0x7fad400b3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:54.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.949+0000 7fad4522d700 1 -- 172.21.15.91:0/3387517199 shutdown_connections 2023-11-24T06:07:54.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:54.949+0000 7fad4522d700 1 -- 172.21.15.91:0/3387517199 wait complete. 2023-11-24T06:07:54.955 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 2 2023-11-24T06:07:56.965 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:07:57.099 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.096+0000 7f8e2ae05700 1 Processor -- start 2023-11-24T06:07:57.099 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.096+0000 7f8e2ae05700 1 -- start start 2023-11-24T06:07:57.101 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.096+0000 7f8e2ae05700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8e2415e1e0 0x7f8e240b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.101 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.096+0000 7f8e2ae05700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f8e2406b400 0x7f8e240b17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.097+0000 7f8e2ae05700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e240b1d20 con 0x7f8e240b5e40 2023-11-24T06:07:57.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.097+0000 7f8e2ae05700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e2406bb80 con 0x7f8e2406b400 2023-11-24T06:07:57.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.097+0000 7f8e2ae05700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2406be20 con 0x7f8e2415e1e0 2023-11-24T06:07:57.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.097+0000 7f8e293a2700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f8e240b5e40 0x7f8e240726a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52466/0 (socket says 172.21.15.91:52466) 2023-11-24T06:07:57.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.097+0000 7f8e293a2700 1 -- 172.21.15.91:0/2570702335 learned_addr learned my addr 172.21.15.91:0/2570702335 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:07:57.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.097+0000 7f8e28ba1700 1 --2- 172.21.15.91:0/2570702335 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8e2415e1e0 0x7f8e240b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.097+0000 7f8e23fff700 1 --2- 172.21.15.91:0/2570702335 >> v2:172.21.15.94:3300/0 conn(0x7f8e2406b400 0x7f8e240b17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.097+0000 7f8e237fe700 1 -- 172.21.15.91:0/2570702335 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580372181 0 0) 0x7f8e240b1d20 con 0x7f8e240b5e40 2023-11-24T06:07:57.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.097+0000 7f8e237fe700 1 -- 172.21.15.91:0/2570702335 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e140035b0 con 0x7f8e240b5e40 2023-11-24T06:07:57.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.097+0000 7f8e28ba1700 1 -- 172.21.15.91:0/2570702335 >> v2:172.21.15.94:3300/0 conn(0x7f8e2406b400 msgr2=0x7f8e240b17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:57.103 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.097+0000 7f8e28ba1700 1 --2- 172.21.15.91:0/2570702335 >> v2:172.21.15.94:3300/0 conn(0x7f8e2406b400 0x7f8e240b17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.103 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.097+0000 7f8e28ba1700 1 -- 172.21.15.91:0/2570702335 >> v1:172.21.15.91:6789/0 conn(0x7f8e240b5e40 legacy=0x7f8e240726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.103 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.097+0000 7f8e28ba1700 1 -- 172.21.15.91:0/2570702335 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e2406c100 con 0x7f8e2415e1e0 2023-11-24T06:07:57.103 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.097+0000 7f8e23fff700 1 --2- 172.21.15.91:0/2570702335 >> v2:172.21.15.94:3300/0 conn(0x7f8e2406b400 0x7f8e240b17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:07:57.103 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.097+0000 7f8e28ba1700 1 --2- 172.21.15.91:0/2570702335 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8e2415e1e0 0x7f8e240b12a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009840 tx=0x7f8e18009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf476498531feefa server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:57.103 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.097+0000 7f8e237fe700 1 -- 172.21.15.91:0/2570702335 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e18011830 con 0x7f8e2415e1e0 2023-11-24T06:07:57.103 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.098+0000 7f8e237fe700 1 -- 172.21.15.91:0/2570702335 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e18011990 con 0x7f8e2415e1e0 2023-11-24T06:07:57.103 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.098+0000 7f8e237fe700 1 -- 172.21.15.91:0/2570702335 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e18011d20 con 0x7f8e2415e1e0 2023-11-24T06:07:57.103 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.098+0000 7f8e2ae05700 1 -- 172.21.15.91:0/2570702335 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8e2415e1e0 msgr2=0x7f8e240b12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.103 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.098+0000 7f8e2ae05700 1 --2- 172.21.15.91:0/2570702335 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8e2415e1e0 0x7f8e240b12a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009840 tx=0x7f8e18009b50 comp rx=0 tx=0).stop 2023-11-24T06:07:57.103 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.098+0000 7f8e2ae05700 1 -- 172.21.15.91:0/2570702335 shutdown_connections 2023-11-24T06:07:57.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.098+0000 7f8e2ae05700 1 --2- 172.21.15.91:0/2570702335 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8e2415e1e0 0x7f8e240b12a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.098+0000 7f8e2ae05700 1 --2- 172.21.15.91:0/2570702335 >> v2:172.21.15.94:3300/0 conn(0x7f8e2406b400 0x7f8e240b17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.098+0000 7f8e2ae05700 1 -- 172.21.15.91:0/2570702335 >> 172.21.15.91:0/2570702335 conn(0x7f8e240b5070 msgr2=0x7f8e240b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:57.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.098+0000 7f8e2ae05700 1 -- 172.21.15.91:0/2570702335 shutdown_connections 2023-11-24T06:07:57.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.098+0000 7f8e2ae05700 1 -- 172.21.15.91:0/2570702335 wait complete. 2023-11-24T06:07:57.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.099+0000 7f8e2ae05700 1 Processor -- start 2023-11-24T06:07:57.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.099+0000 7f8e2ae05700 1 -- start start 2023-11-24T06:07:57.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.100+0000 7f8e2ae05700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8e2406b400 0x7f8e2414da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.100+0000 7f8e2ae05700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e240b5e40 0x7f8e2406e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.100+0000 7f8e2ae05700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e2415e1e0 0x7f8e241f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.100+0000 7f8e2ae05700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e241f3480 con 0x7f8e2415e1e0 2023-11-24T06:07:57.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.100+0000 7f8e2ae05700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2409adb0 con 0x7f8e240b5e40 2023-11-24T06:07:57.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.100+0000 7f8e2ae05700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2409b050 con 0x7f8e2406b400 2023-11-24T06:07:57.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.100+0000 7f8e293a2700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e2415e1e0 0x7f8e241f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.100+0000 7f8e293a2700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e2415e1e0 0x7f8e241f2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57538/0 (socket says 172.21.15.91:57538) 2023-11-24T06:07:57.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.100+0000 7f8e293a2700 1 -- 172.21.15.91:0/1704115679 learned_addr learned my addr 172.21.15.91:0/1704115679 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:57.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.101+0000 7f8e28ba1700 1 --2- 172.21.15.91:0/1704115679 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8e2406b400 0x7f8e2414da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.101+0000 7f8e23fff700 1 --2- 172.21.15.91:0/1704115679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e240b5e40 0x7f8e2406e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.101+0000 7f8e293a2700 1 -- 172.21.15.91:0/1704115679 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8e2406b400 msgr2=0x7f8e2414da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.101+0000 7f8e293a2700 1 --2- 172.21.15.91:0/1704115679 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8e2406b400 0x7f8e2414da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.101+0000 7f8e293a2700 1 -- 172.21.15.91:0/1704115679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e240b5e40 msgr2=0x7f8e2406e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.101+0000 7f8e293a2700 1 --2- 172.21.15.91:0/1704115679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e240b5e40 0x7f8e2406e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.101+0000 7f8e293a2700 1 -- 172.21.15.91:0/1704115679 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e100094a0 con 0x7f8e2415e1e0 2023-11-24T06:07:57.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.101+0000 7f8e23fff700 1 --2- 172.21.15.91:0/1704115679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e240b5e40 0x7f8e2406e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:07:57.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.101+0000 7f8e293a2700 1 --2- 172.21.15.91:0/1704115679 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e2415e1e0 0x7f8e241f2f40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00cb40 tx=0x7f8e1c00ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:57.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.101+0000 7f8e217fa700 1 -- 172.21.15.91:0/1704115679 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c00aaa0 con 0x7f8e2415e1e0 2023-11-24T06:07:57.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.101+0000 7f8e217fa700 1 -- 172.21.15.91:0/1704115679 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1c00ac00 con 0x7f8e2415e1e0 2023-11-24T06:07:57.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.101+0000 7f8e217fa700 1 -- 172.21.15.91:0/1704115679 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c012400 con 0x7f8e2415e1e0 2023-11-24T06:07:57.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.101+0000 7f8e2ae05700 1 -- 172.21.15.91:0/1704115679 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e180094a0 con 0x7f8e2415e1e0 2023-11-24T06:07:57.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.101+0000 7f8e2ae05700 1 -- 172.21.15.91:0/1704115679 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e2409cd60 con 0x7f8e2415e1e0 2023-11-24T06:07:57.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.102+0000 7f8e2ae05700 1 -- 172.21.15.91:0/1704115679 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e2409cbc0 con 0x7f8e2415e1e0 2023-11-24T06:07:57.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.102+0000 7f8e217fa700 1 -- 172.21.15.91:0/1704115679 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8e1c01b3b0 con 0x7f8e2415e1e0 2023-11-24T06:07:57.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.103+0000 7f8e217fa700 1 --2- 172.21.15.91:0/1704115679 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8e0806aed0 0x7f8e0806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.103+0000 7f8e28ba1700 1 --2- 172.21.15.91:0/1704115679 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8e0806aed0 0x7f8e0806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.103+0000 7f8e217fa700 1 -- 172.21.15.91:0/1704115679 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8e1c088f60 con 0x7f8e2415e1e0 2023-11-24T06:07:57.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.103+0000 7f8e28ba1700 1 --2- 172.21.15.91:0/1704115679 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8e0806aed0 0x7f8e0806d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009fd0 tx=0x7f8e18008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:57.110 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.107+0000 7f8e217fa700 1 -- 172.21.15.91:0/1704115679 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e2409cbc0 con 0x7f8e2415e1e0 2023-11-24T06:07:57.341 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.338+0000 7f8e2ae05700 1 -- 172.21.15.91:0/1704115679 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e2409cbc0 con 0x7f8e2415e1e0 2023-11-24T06:07:57.341 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.338+0000 7f8e217fa700 1 -- 172.21.15.91:0/1704115679 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3694 (secure 0 0 0) 0x7f8e1c017e80 con 0x7f8e2415e1e0 2023-11-24T06:07:57.342 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:57.342 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1329219967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1329219967},{"type":"v1","addr":"172.21.15.94:6834","nonce":1329219967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3439524864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3439524864},{"type":"v1","addr":"172.21.15.94:6835","nonce":3439524864}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6836/3390368219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6834","nonce":3390368219},{"type":"v1","addr":"172.21.15.91:6836","nonce":3390368219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6837/1644079645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":1644079645},{"type":"v1","addr":"172.21.15.91:6837","nonce":1644079645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-11-24T06:07:57.343 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.340+0000 7f8e2ae05700 1 -- 172.21.15.91:0/1704115679 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8e0806aed0 msgr2=0x7f8e0806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.340+0000 7f8e2ae05700 1 --2- 172.21.15.91:0/1704115679 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8e0806aed0 0x7f8e0806d390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009fd0 tx=0x7f8e18008030 comp rx=0 tx=0).stop 2023-11-24T06:07:57.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.340+0000 7f8e2ae05700 1 -- 172.21.15.91:0/1704115679 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e2415e1e0 msgr2=0x7f8e241f2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.340+0000 7f8e2ae05700 1 --2- 172.21.15.91:0/1704115679 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e2415e1e0 0x7f8e241f2f40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00cb40 tx=0x7f8e1c00ce50 comp rx=0 tx=0).stop 2023-11-24T06:07:57.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.340+0000 7f8e2ae05700 1 -- 172.21.15.91:0/1704115679 shutdown_connections 2023-11-24T06:07:57.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.340+0000 7f8e2ae05700 1 --2- 172.21.15.91:0/1704115679 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8e0806aed0 0x7f8e0806d390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.340+0000 7f8e2ae05700 1 --2- 172.21.15.91:0/1704115679 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e2415e1e0 0x7f8e241f2f40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.340+0000 7f8e2ae05700 1 --2- 172.21.15.91:0/1704115679 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8e2406b400 0x7f8e2414da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.340+0000 7f8e2ae05700 1 --2- 172.21.15.91:0/1704115679 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e240b5e40 0x7f8e2406e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.340+0000 7f8e2ae05700 1 -- 172.21.15.91:0/1704115679 >> 172.21.15.91:0/1704115679 conn(0x7f8e240b5070 msgr2=0x7f8e240726a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:57.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.341+0000 7f8e2ae05700 1 -- 172.21.15.91:0/1704115679 shutdown_connections 2023-11-24T06:07:57.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.341+0000 7f8e2ae05700 1 -- 172.21.15.91:0/1704115679 wait complete. 2023-11-24T06:07:57.345 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 3 2023-11-24T06:07:57.355 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:07:57.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.473+0000 7f538feb7700 1 Processor -- start 2023-11-24T06:07:57.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.474+0000 7f538feb7700 1 -- start start 2023-11-24T06:07:57.479 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.474+0000 7f538feb7700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f53880cf2b0 0x7f538815c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.479 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.474+0000 7f538feb7700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53880d0a90 con 0x7f53880ce5c0 2023-11-24T06:07:57.479 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.474+0000 7f538feb7700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538806ee10 con 0x7f538815bde0 2023-11-24T06:07:57.479 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.474+0000 7f538feb7700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f538806f0f0 con 0x7f53880cf2b0 2023-11-24T06:07:57.479 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.475+0000 7f538e454700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f53880ce5c0 0x7f538806fbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52480/0 (socket says 172.21.15.91:52480) 2023-11-24T06:07:57.480 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.475+0000 7f538e454700 1 -- 172.21.15.91:0/2209884268 learned_addr learned my addr 172.21.15.91:0/2209884268 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:07:57.480 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.475+0000 7f538d452700 1 --2- 172.21.15.91:0/2209884268 >> v2:172.21.15.94:3300/0 conn(0x7f53880cf2b0 0x7f538815c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.480 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.475+0000 7f538cc51700 1 -- 172.21.15.91:0/2209884268 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823684766 0 0) 0x7f53880d0a90 con 0x7f53880ce5c0 2023-11-24T06:07:57.480 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.475+0000 7f538cc51700 1 -- 172.21.15.91:0/2209884268 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53740035b0 con 0x7f53880ce5c0 2023-11-24T06:07:57.480 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.475+0000 7f538d452700 1 -- 172.21.15.91:0/2209884268 >> v1:172.21.15.94:6789/0 conn(0x7f538815bde0 legacy=0x7f53880d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.480 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.475+0000 7f538d452700 1 -- 172.21.15.91:0/2209884268 >> v1:172.21.15.91:6789/0 conn(0x7f53880ce5c0 legacy=0x7f538806fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.480 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.475+0000 7f538d452700 1 -- 172.21.15.91:0/2209884268 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538806f3d0 con 0x7f53880cf2b0 2023-11-24T06:07:57.480 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.475+0000 7f538d452700 1 --2- 172.21.15.91:0/2209884268 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53880cf2b0 0x7f538815c1c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5384009840 tx=0x7f5384009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c7a0c824fb01cbd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:57.480 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.475+0000 7f538cc51700 1 -- 172.21.15.91:0/2209884268 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384011830 con 0x7f53880cf2b0 2023-11-24T06:07:57.480 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.475+0000 7f538cc51700 1 -- 172.21.15.91:0/2209884268 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5384011990 con 0x7f53880cf2b0 2023-11-24T06:07:57.480 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.475+0000 7f538cc51700 1 -- 172.21.15.91:0/2209884268 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384011d20 con 0x7f53880cf2b0 2023-11-24T06:07:57.481 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.476+0000 7f538feb7700 1 -- 172.21.15.91:0/2209884268 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53880cf2b0 msgr2=0x7f538815c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.481 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.476+0000 7f538feb7700 1 --2- 172.21.15.91:0/2209884268 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53880cf2b0 0x7f538815c1c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5384009840 tx=0x7f5384009b50 comp rx=0 tx=0).stop 2023-11-24T06:07:57.481 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.476+0000 7f538feb7700 1 -- 172.21.15.91:0/2209884268 shutdown_connections 2023-11-24T06:07:57.481 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.476+0000 7f538feb7700 1 --2- 172.21.15.91:0/2209884268 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53880cf2b0 0x7f538815c1c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.481 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.476+0000 7f538feb7700 1 -- 172.21.15.91:0/2209884268 >> 172.21.15.91:0/2209884268 conn(0x7f53880b3720 msgr2=0x7f53880b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:57.481 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.476+0000 7f538feb7700 1 -- 172.21.15.91:0/2209884268 shutdown_connections 2023-11-24T06:07:57.481 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.476+0000 7f538feb7700 1 -- 172.21.15.91:0/2209884268 wait complete. 2023-11-24T06:07:57.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.477+0000 7f538feb7700 1 Processor -- start 2023-11-24T06:07:57.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.477+0000 7f538feb7700 1 -- start start 2023-11-24T06:07:57.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538feb7700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53880ce5c0 0x7f538815a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538feb7700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53880cf2b0 0x7f538806d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538feb7700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f538815bde0 0x7f53880c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538feb7700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53880b6520 con 0x7f53880cf2b0 2023-11-24T06:07:57.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538feb7700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53880c4be0 con 0x7f53880ce5c0 2023-11-24T06:07:57.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538feb7700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538806dc90 con 0x7f538815bde0 2023-11-24T06:07:57.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538d452700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53880cf2b0 0x7f538806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538d452700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53880cf2b0 0x7f538806d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57550/0 (socket says 172.21.15.91:57550) 2023-11-24T06:07:57.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538d452700 1 -- 172.21.15.91:0/52004785 learned_addr learned my addr 172.21.15.91:0/52004785 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:57.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538dc53700 1 --2- 172.21.15.91:0/52004785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53880ce5c0 0x7f538815a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538e454700 1 --2- 172.21.15.91:0/52004785 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f538815bde0 0x7f53880c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538d452700 1 -- 172.21.15.91:0/52004785 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f538815bde0 msgr2=0x7f53880c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538d452700 1 --2- 172.21.15.91:0/52004785 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f538815bde0 0x7f53880c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538d452700 1 -- 172.21.15.91:0/52004785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53880ce5c0 msgr2=0x7f538815a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538d452700 1 --2- 172.21.15.91:0/52004785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53880ce5c0 0x7f538815a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538d452700 1 -- 172.21.15.91:0/52004785 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53840094a0 con 0x7f53880cf2b0 2023-11-24T06:07:57.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.478+0000 7f538d452700 1 --2- 172.21.15.91:0/52004785 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53880cf2b0 0x7f538806d590 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5384009fd0 tx=0x7f5384004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:57.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.479+0000 7f537e7fc700 1 -- 172.21.15.91:0/52004785 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53840038a0 con 0x7f53880cf2b0 2023-11-24T06:07:57.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.479+0000 7f537e7fc700 1 -- 172.21.15.91:0/52004785 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538401f070 con 0x7f53880cf2b0 2023-11-24T06:07:57.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.479+0000 7f537e7fc700 1 -- 172.21.15.91:0/52004785 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384003c40 con 0x7f53880cf2b0 2023-11-24T06:07:57.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.479+0000 7f538feb7700 1 -- 172.21.15.91:0/52004785 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f538806df70 con 0x7f53880cf2b0 2023-11-24T06:07:57.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.479+0000 7f538feb7700 1 -- 172.21.15.91:0/52004785 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53880b5e60 con 0x7f53880cf2b0 2023-11-24T06:07:57.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.480+0000 7f538feb7700 1 -- 172.21.15.91:0/52004785 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53880b6020 con 0x7f53880cf2b0 2023-11-24T06:07:57.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.480+0000 7f537e7fc700 1 -- 172.21.15.91:0/52004785 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5384026480 con 0x7f53880cf2b0 2023-11-24T06:07:57.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.480+0000 7f537e7fc700 1 --2- 172.21.15.91:0/52004785 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f536c06aed0 0x7f536c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.480+0000 7f537e7fc700 1 -- 172.21.15.91:0/52004785 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5384021070 con 0x7f53880cf2b0 2023-11-24T06:07:57.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.480+0000 7f538dc53700 1 --2- 172.21.15.91:0/52004785 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f536c06aed0 0x7f536c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.481+0000 7f538dc53700 1 --2- 172.21.15.91:0/52004785 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f536c06aed0 0x7f536c06d390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5378000f60 tx=0x7f5378009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:57.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.485+0000 7f537e7fc700 1 -- 172.21.15.91:0/52004785 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5384056070 con 0x7f53880cf2b0 2023-11-24T06:07:57.679 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.676+0000 7f538feb7700 1 -- 172.21.15.91:0/52004785 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5388068670 con 0x7f53880cf2b0 2023-11-24T06:07:57.680 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.677+0000 7f537e7fc700 1 -- 172.21.15.91:0/52004785 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12766 (secure 0 0 0) 0x7f538402b090 con 0x7f53880cf2b0 2023-11-24T06:07:57.680 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:57.680 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":16,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:07:49.871796+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:07:57.682 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.679+0000 7f538feb7700 1 -- 172.21.15.91:0/52004785 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f536c06aed0 msgr2=0x7f536c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.682 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.679+0000 7f538feb7700 1 --2- 172.21.15.91:0/52004785 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f536c06aed0 0x7f536c06d390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5378000f60 tx=0x7f5378009040 comp rx=0 tx=0).stop 2023-11-24T06:07:57.682 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.679+0000 7f538feb7700 1 -- 172.21.15.91:0/52004785 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53880cf2b0 msgr2=0x7f538806d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.682 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.679+0000 7f538feb7700 1 --2- 172.21.15.91:0/52004785 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53880cf2b0 0x7f538806d590 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5384009fd0 tx=0x7f5384004750 comp rx=0 tx=0).stop 2023-11-24T06:07:57.683 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.679+0000 7f538feb7700 1 -- 172.21.15.91:0/52004785 shutdown_connections 2023-11-24T06:07:57.683 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.679+0000 7f538feb7700 1 --2- 172.21.15.91:0/52004785 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f536c06aed0 0x7f536c06d390 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.683 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.679+0000 7f538feb7700 1 --2- 172.21.15.91:0/52004785 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53880ce5c0 0x7f538815a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.683 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.679+0000 7f538feb7700 1 --2- 172.21.15.91:0/52004785 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f538815bde0 0x7f53880c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.683 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.679+0000 7f538feb7700 1 --2- 172.21.15.91:0/52004785 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53880cf2b0 0x7f538806d590 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.683 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.679+0000 7f538feb7700 1 -- 172.21.15.91:0/52004785 >> 172.21.15.91:0/52004785 conn(0x7f53880b3720 msgr2=0x7f5388070800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:57.683 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.679+0000 7f538feb7700 1 -- 172.21.15.91:0/52004785 shutdown_connections 2023-11-24T06:07:57.683 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.679+0000 7f538feb7700 1 -- 172.21.15.91:0/52004785 wait complete. 2023-11-24T06:07:57.694 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:07:57.694 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:07:57.818 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.815+0000 7f653e8c3700 1 Processor -- start 2023-11-24T06:07:57.818 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.815+0000 7f653e8c3700 1 -- start start 2023-11-24T06:07:57.820 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f653e8c3700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f653806b620 0x7f65380b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.820 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f653e8c3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f653806ee60 0x7f65380b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f653e8c3700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6538071f50 con 0x7f65380b5ab0 2023-11-24T06:07:57.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f653e8c3700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6538072230 con 0x7f653806b620 2023-11-24T06:07:57.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f653e8c3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6538072510 con 0x7f653806ee60 2023-11-24T06:07:57.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f6537fff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f653806b620 0x7f65380b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f65377fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f65380b5ab0 0x7f6538161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52492/0 (socket says 172.21.15.91:52492) 2023-11-24T06:07:57.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f65377fe700 1 -- 172.21.15.91:0/4265720108 learned_addr learned my addr 172.21.15.91:0/4265720108 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:07:57.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f653ce60700 1 --2- 172.21.15.91:0/4265720108 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f653806ee60 0x7f65380b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f6537fff700 1 -- 172.21.15.91:0/4265720108 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f653806ee60 msgr2=0x7f65380b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:57.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f6537fff700 1 --2- 172.21.15.91:0/4265720108 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f653806ee60 0x7f65380b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f6537fff700 1 -- 172.21.15.91:0/4265720108 >> v1:172.21.15.91:6789/0 conn(0x7f65380b5ab0 legacy=0x7f6538161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f6537fff700 1 -- 172.21.15.91:0/4265720108 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65380727f0 con 0x7f653806b620 2023-11-24T06:07:57.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f6536ffd700 1 -- 172.21.15.91:0/4265720108 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252573710 0 0) 0x7f6538071f50 con 0x7f65380b5ab0 2023-11-24T06:07:57.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f653ce60700 1 --2- 172.21.15.91:0/4265720108 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f653806ee60 0x7f65380b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:07:57.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f6537fff700 1 --2- 172.21.15.91:0/4265720108 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f653806b620 0x7f65380b3a40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6528009840 tx=0x7f6528009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d6d754964c2b9b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:57.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f6536ffd700 1 -- 172.21.15.91:0/4265720108 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6528011830 con 0x7f653806b620 2023-11-24T06:07:57.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f6536ffd700 1 -- 172.21.15.91:0/4265720108 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6528011990 con 0x7f653806b620 2023-11-24T06:07:57.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.816+0000 7f6536ffd700 1 -- 172.21.15.91:0/4265720108 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6528011d20 con 0x7f653806b620 2023-11-24T06:07:57.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.817+0000 7f653e8c3700 1 -- 172.21.15.91:0/4265720108 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f653806b620 msgr2=0x7f65380b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.817+0000 7f653e8c3700 1 --2- 172.21.15.91:0/4265720108 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f653806b620 0x7f65380b3a40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6528009840 tx=0x7f6528009b50 comp rx=0 tx=0).stop 2023-11-24T06:07:57.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.817+0000 7f653e8c3700 1 -- 172.21.15.91:0/4265720108 shutdown_connections 2023-11-24T06:07:57.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.817+0000 7f653e8c3700 1 --2- 172.21.15.91:0/4265720108 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f653806ee60 0x7f65380b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.817+0000 7f653e8c3700 1 --2- 172.21.15.91:0/4265720108 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f653806b620 0x7f65380b3a40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.817+0000 7f653e8c3700 1 -- 172.21.15.91:0/4265720108 >> 172.21.15.91:0/4265720108 conn(0x7f65380b2740 msgr2=0x7f65380b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:57.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.817+0000 7f653e8c3700 1 -- 172.21.15.91:0/4265720108 shutdown_connections 2023-11-24T06:07:57.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.817+0000 7f653e8c3700 1 -- 172.21.15.91:0/4265720108 wait complete. 2023-11-24T06:07:57.825 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.818+0000 7f653e8c3700 1 Processor -- start 2023-11-24T06:07:57.825 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.818+0000 7f653e8c3700 1 -- start start 2023-11-24T06:07:57.825 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.819+0000 7f653e8c3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f653806b620 0x7f65381e8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.819+0000 7f653e8c3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f653806ee60 0x7f653813e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.819+0000 7f653e8c3700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f65380b5ab0 0x7f65381417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.819+0000 7f653e8c3700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65381e77f0 con 0x7f65380b5ab0 2023-11-24T06:07:57.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.819+0000 7f653e8c3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65381e7ad0 con 0x7f653806b620 2023-11-24T06:07:57.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.819+0000 7f653e8c3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65381e7db0 con 0x7f653806ee60 2023-11-24T06:07:57.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.819+0000 7f653ce60700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f65380b5ab0 0x7f65381417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.819+0000 7f653ce60700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f65380b5ab0 0x7f65381417d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57566/0 (socket says 172.21.15.91:57566) 2023-11-24T06:07:57.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.819+0000 7f653ce60700 1 -- 172.21.15.91:0/3289747537 learned_addr learned my addr 172.21.15.91:0/3289747537 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:57.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.819+0000 7f6537fff700 1 --2- 172.21.15.91:0/3289747537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f653806b620 0x7f65381e8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.819+0000 7f65377fe700 1 --2- 172.21.15.91:0/3289747537 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f653806ee60 0x7f653813e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.819+0000 7f653ce60700 1 -- 172.21.15.91:0/3289747537 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f653806ee60 msgr2=0x7f653813e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.827 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.819+0000 7f653ce60700 1 --2- 172.21.15.91:0/3289747537 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f653806ee60 0x7f653813e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.827 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.819+0000 7f653ce60700 1 -- 172.21.15.91:0/3289747537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f653806b620 msgr2=0x7f65381e8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:57.827 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.819+0000 7f653ce60700 1 --2- 172.21.15.91:0/3289747537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f653806b620 0x7f65381e8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:57.827 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.819+0000 7f653ce60700 1 -- 172.21.15.91:0/3289747537 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65280094a0 con 0x7f65380b5ab0 2023-11-24T06:07:57.827 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.820+0000 7f653ce60700 1 --2- 172.21.15.91:0/3289747537 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f65380b5ab0 0x7f65381417d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f653000cea0 tx=0x7f653000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:57.827 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.820+0000 7f6534ff9700 1 -- 172.21.15.91:0/3289747537 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653000ae50 con 0x7f65380b5ab0 2023-11-24T06:07:57.827 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.820+0000 7f6534ff9700 1 -- 172.21.15.91:0/3289747537 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6530003a20 con 0x7f65380b5ab0 2023-11-24T06:07:57.827 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.820+0000 7f6534ff9700 1 -- 172.21.15.91:0/3289747537 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530003dd0 con 0x7f65380b5ab0 2023-11-24T06:07:57.827 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.820+0000 7f653e8c3700 1 -- 172.21.15.91:0/3289747537 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65381e8090 con 0x7f65380b5ab0 2023-11-24T06:07:57.827 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.820+0000 7f653e8c3700 1 -- 172.21.15.91:0/3289747537 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6538140310 con 0x7f65380b5ab0 2023-11-24T06:07:57.827 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.821+0000 7f653e8c3700 1 -- 172.21.15.91:0/3289747537 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6538075480 con 0x7f65380b5ab0 2023-11-24T06:07:57.828 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.821+0000 7f6534ff9700 1 -- 172.21.15.91:0/3289747537 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6530003f30 con 0x7f65380b5ab0 2023-11-24T06:07:57.828 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.822+0000 7f6534ff9700 1 --2- 172.21.15.91:0/3289747537 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f652406aed0 0x7f652406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:57.828 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.822+0000 7f6537fff700 1 --2- 172.21.15.91:0/3289747537 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f652406aed0 0x7f652406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:57.828 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.822+0000 7f6534ff9700 1 -- 172.21.15.91:0/3289747537 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f65300896d0 con 0x7f65380b5ab0 2023-11-24T06:07:57.828 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.822+0000 7f6537fff700 1 --2- 172.21.15.91:0/3289747537 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f652406aed0 0x7f652406d390 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f65280090b0 tx=0x7f6528008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:57.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:57.826+0000 7f6534ff9700 1 -- 172.21.15.91:0/3289747537 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6530054ad0 con 0x7f65380b5ab0 2023-11-24T06:07:58.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.014+0000 7f653e8c3700 1 -- 172.21.15.91:0/3289747537 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f65381e5630 con 0x7f65380b5ab0 2023-11-24T06:07:58.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.014+0000 7f6534ff9700 1 -- 172.21.15.91:0/3289747537 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f65300580f0 con 0x7f65380b5ab0 2023-11-24T06:07:58.017 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:07:58.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.016+0000 7f653e8c3700 1 -- 172.21.15.91:0/3289747537 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f652406aed0 msgr2=0x7f652406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:58.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.016+0000 7f653e8c3700 1 --2- 172.21.15.91:0/3289747537 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f652406aed0 0x7f652406d390 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f65280090b0 tx=0x7f6528008030 comp rx=0 tx=0).stop 2023-11-24T06:07:58.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.016+0000 7f653e8c3700 1 -- 172.21.15.91:0/3289747537 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f65380b5ab0 msgr2=0x7f65381417d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:58.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.016+0000 7f653e8c3700 1 --2- 172.21.15.91:0/3289747537 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f65380b5ab0 0x7f65381417d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f653000cea0 tx=0x7f653000cf80 comp rx=0 tx=0).stop 2023-11-24T06:07:58.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.016+0000 7f653e8c3700 1 -- 172.21.15.91:0/3289747537 shutdown_connections 2023-11-24T06:07:58.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.016+0000 7f653e8c3700 1 --2- 172.21.15.91:0/3289747537 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f652406aed0 0x7f652406d390 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.016+0000 7f653e8c3700 1 --2- 172.21.15.91:0/3289747537 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f653806b620 0x7f65381e8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.016+0000 7f653e8c3700 1 --2- 172.21.15.91:0/3289747537 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f65380b5ab0 0x7f65381417d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.016+0000 7f653e8c3700 1 --2- 172.21.15.91:0/3289747537 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f653806ee60 0x7f653813e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.016+0000 7f653e8c3700 1 -- 172.21.15.91:0/3289747537 >> 172.21.15.91:0/3289747537 conn(0x7f65380b2740 msgr2=0x7f65380aee80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:58.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.017+0000 7f653e8c3700 1 -- 172.21.15.91:0/3289747537 shutdown_connections 2023-11-24T06:07:58.021 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.017+0000 7f653e8c3700 1 -- 172.21.15.91:0/3289747537 wait complete. 2023-11-24T06:07:58.032 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T06:07:58.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.148+0000 7fad39812700 1 Processor -- start 2023-11-24T06:07:58.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.148+0000 7fad39812700 1 -- start start 2023-11-24T06:07:58.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.149+0000 7fad39812700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad340d07f0 0x7fad340b4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:58.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.149+0000 7fad39812700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fad340c6070 0x7fad340c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:58.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.149+0000 7fad39812700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad340c5be0 con 0x7fad340cfd00 2023-11-24T06:07:58.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.149+0000 7fad39812700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad340cdad0 con 0x7fad340c6070 2023-11-24T06:07:58.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.149+0000 7fad39812700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad340cd230 con 0x7fad340d07f0 2023-11-24T06:07:58.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.149+0000 7fad32ffd700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fad340cfd00 0x7fad340d00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52504/0 (socket says 172.21.15.91:52504) 2023-11-24T06:07:58.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.149+0000 7fad32ffd700 1 -- 172.21.15.91:0/523163002 learned_addr learned my addr 172.21.15.91:0/523163002 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:07:58.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.149+0000 7fad337fe700 1 --2- 172.21.15.91:0/523163002 >> v2:172.21.15.94:3300/0 conn(0x7fad340c6070 0x7fad340c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:58.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.149+0000 7fad327fc700 1 --2- 172.21.15.91:0/523163002 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad340d07f0 0x7fad340b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:58.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.149+0000 7fad31ffb700 1 -- 172.21.15.91:0/523163002 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797034982 0 0) 0x7fad340c5be0 con 0x7fad340cfd00 2023-11-24T06:07:58.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.149+0000 7fad31ffb700 1 -- 172.21.15.91:0/523163002 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad1c0035b0 con 0x7fad340cfd00 2023-11-24T06:07:58.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 -- 172.21.15.91:0/523163002 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 839704712 0 0) 0x7fad1c0035b0 con 0x7fad340cfd00 2023-11-24T06:07:58.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 -- 172.21.15.91:0/523163002 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad340d07f0 msgr2=0x7fad340b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:58.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 --2- 172.21.15.91:0/523163002 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad340d07f0 0x7fad340b4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 -- 172.21.15.91:0/523163002 >> v2:172.21.15.94:3300/0 conn(0x7fad340c6070 msgr2=0x7fad340c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:58.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 --2- 172.21.15.91:0/523163002 >> v2:172.21.15.94:3300/0 conn(0x7fad340c6070 0x7fad340c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 -- 172.21.15.91:0/523163002 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad340cd510 con 0x7fad340cfd00 2023-11-24T06:07:58.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 -- 172.21.15.91:0/523163002 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7fad28001670 con 0x7fad340cfd00 2023-11-24T06:07:58.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 -- 172.21.15.91:0/523163002 >> v1:172.21.15.91:6789/0 conn(0x7fad340cfd00 legacy=0x7fad340d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:58.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 --2- 172.21.15.91:0/523163002 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad1c003a00 0x7fad1c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:58.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 --2- 172.21.15.91:0/523163002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad1c004850 0x7fad1c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:58.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 --2- 172.21.15.91:0/523163002 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad340d07f0 0x7fad1c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:58.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 -- 172.21.15.91:0/523163002 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad340cdad0 con 0x7fad1c003a00 2023-11-24T06:07:58.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 -- 172.21.15.91:0/523163002 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad340cd230 con 0x7fad1c004850 2023-11-24T06:07:58.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 -- 172.21.15.91:0/523163002 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1c00ba10 con 0x7fad340d07f0 2023-11-24T06:07:58.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad32ffd700 1 --2- 172.21.15.91:0/523163002 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad1c003a00 0x7fad1c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:58.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad327fc700 1 --2- 172.21.15.91:0/523163002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad1c004850 0x7fad1c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:58.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad337fe700 1 --2- 172.21.15.91:0/523163002 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad340d07f0 0x7fad1c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:58.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad32ffd700 1 -- 172.21.15.91:0/523163002 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad340d07f0 msgr2=0x7fad1c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:58.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad32ffd700 1 --2- 172.21.15.91:0/523163002 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad340d07f0 0x7fad1c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad32ffd700 1 -- 172.21.15.91:0/523163002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad1c004850 msgr2=0x7fad1c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:58.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad32ffd700 1 --2- 172.21.15.91:0/523163002 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad1c004850 0x7fad1c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad32ffd700 1 -- 172.21.15.91:0/523163002 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1c0035b0 con 0x7fad1c003a00 2023-11-24T06:07:58.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad32ffd700 1 --2- 172.21.15.91:0/523163002 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad1c003a00 0x7fad1c003e20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fad2800c320 tx=0x7fad1c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a86595e8a91dfbc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:58.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 -- 172.21.15.91:0/523163002 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2800ce60 con 0x7fad1c003a00 2023-11-24T06:07:58.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 -- 172.21.15.91:0/523163002 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad28005960 con 0x7fad1c003a00 2023-11-24T06:07:58.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.150+0000 7fad31ffb700 1 -- 172.21.15.91:0/523163002 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad28005d10 con 0x7fad1c003a00 2023-11-24T06:07:58.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.151+0000 7fad39812700 1 -- 172.21.15.91:0/523163002 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad1c003a00 msgr2=0x7fad1c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:58.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.151+0000 7fad39812700 1 --2- 172.21.15.91:0/523163002 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad1c003a00 0x7fad1c003e20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fad2800c320 tx=0x7fad1c005880 comp rx=0 tx=0).stop 2023-11-24T06:07:58.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.151+0000 7fad337fe700 1 -- 172.21.15.91:0/523163002 reap_dead start 2023-11-24T06:07:58.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.151+0000 7fad39812700 1 -- 172.21.15.91:0/523163002 shutdown_connections 2023-11-24T06:07:58.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.151+0000 7fad39812700 1 -- 172.21.15.91:0/523163002 >> 172.21.15.91:0/523163002 conn(0x7fad340b3870 msgr2=0x7fad340b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:58.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.151+0000 7fad39812700 1 -- 172.21.15.91:0/523163002 shutdown_connections 2023-11-24T06:07:58.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.151+0000 7fad39812700 1 -- 172.21.15.91:0/523163002 wait complete. 2023-11-24T06:07:58.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.152+0000 7fad39812700 1 Processor -- start 2023-11-24T06:07:58.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.152+0000 7fad39812700 1 -- start start 2023-11-24T06:07:58.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.153+0000 7fad39812700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad1c003a00 0x7fad34159480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:58.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.153+0000 7fad39812700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad340b5110 0x7fad3406fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:58.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.153+0000 7fad39812700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad341f2ba0 0x7fad341f3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:58.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.153+0000 7fad39812700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad340ca300 con 0x7fad340b5110 2023-11-24T06:07:58.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.153+0000 7fad39812700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad340c9470 con 0x7fad1c003a00 2023-11-24T06:07:58.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.153+0000 7fad39812700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad341599c0 con 0x7fad341f2ba0 2023-11-24T06:07:58.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.153+0000 7fad327fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad340b5110 0x7fad3406fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:58.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.153+0000 7fad327fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad340b5110 0x7fad3406fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57584/0 (socket says 172.21.15.91:57584) 2023-11-24T06:07:58.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.153+0000 7fad327fc700 1 -- 172.21.15.91:0/3049243926 learned_addr learned my addr 172.21.15.91:0/3049243926 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:58.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.153+0000 7fad337fe700 1 --2- 172.21.15.91:0/3049243926 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad341f2ba0 0x7fad341f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:58.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.153+0000 7fad32ffd700 1 --2- 172.21.15.91:0/3049243926 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad1c003a00 0x7fad34159480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:58.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.153+0000 7fad327fc700 1 -- 172.21.15.91:0/3049243926 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad341f2ba0 msgr2=0x7fad341f3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:58.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.153+0000 7fad327fc700 1 --2- 172.21.15.91:0/3049243926 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad341f2ba0 0x7fad341f3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.153+0000 7fad327fc700 1 -- 172.21.15.91:0/3049243926 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad1c003a00 msgr2=0x7fad34159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:58.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.153+0000 7fad327fc700 1 --2- 172.21.15.91:0/3049243926 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad1c003a00 0x7fad34159480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.154+0000 7fad327fc700 1 -- 172.21.15.91:0/3049243926 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad341f3540 con 0x7fad340b5110 2023-11-24T06:07:58.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.154+0000 7fad32ffd700 1 --2- 172.21.15.91:0/3049243926 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad1c003a00 0x7fad34159480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:07:58.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.154+0000 7fad327fc700 1 --2- 172.21.15.91:0/3049243926 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad340b5110 0x7fad3406fa60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fad240119d0 tx=0x7fad24011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:58.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.154+0000 7fad1bfff700 1 -- 172.21.15.91:0/3049243926 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2400f940 con 0x7fad340b5110 2023-11-24T06:07:58.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.154+0000 7fad1bfff700 1 -- 172.21.15.91:0/3049243926 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2400faa0 con 0x7fad340b5110 2023-11-24T06:07:58.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.154+0000 7fad39812700 1 -- 172.21.15.91:0/3049243926 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad340c9680 con 0x7fad340b5110 2023-11-24T06:07:58.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.154+0000 7fad1bfff700 1 -- 172.21.15.91:0/3049243926 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2400fe30 con 0x7fad340b5110 2023-11-24T06:07:58.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.154+0000 7fad39812700 1 -- 172.21.15.91:0/3049243926 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad340c9c40 con 0x7fad340b5110 2023-11-24T06:07:58.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.155+0000 7fad39812700 1 -- 172.21.15.91:0/3049243926 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad34067f60 con 0x7fad340b5110 2023-11-24T06:07:58.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.155+0000 7fad1bfff700 1 -- 172.21.15.91:0/3049243926 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fad24014440 con 0x7fad340b5110 2023-11-24T06:07:58.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.155+0000 7fad1bfff700 1 --2- 172.21.15.91:0/3049243926 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad2006aed0 0x7fad2006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:58.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.156+0000 7fad32ffd700 1 --2- 172.21.15.91:0/3049243926 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad2006aed0 0x7fad2006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:58.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.156+0000 7fad1bfff700 1 -- 172.21.15.91:0/3049243926 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fad24089930 con 0x7fad340b5110 2023-11-24T06:07:58.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.156+0000 7fad32ffd700 1 --2- 172.21.15.91:0/3049243926 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad2006aed0 0x7fad2006d390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fad28015040 tx=0x7fad2800a000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:58.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.160+0000 7fad1bfff700 1 -- 172.21.15.91:0/3049243926 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad24054d30 con 0x7fad340b5110 2023-11-24T06:07:58.354 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.351+0000 7fad39812700 1 -- 172.21.15.91:0/3049243926 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fad340ce470 con 0x7fad340b5110 2023-11-24T06:07:58.354 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.351+0000 7fad1bfff700 1 -- 172.21.15.91:0/3049243926 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7fad24058350 con 0x7fad340b5110 2023-11-24T06:07:58.355 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:07:58.355 INFO:teuthology.orchestra.run.smithi091.stdout:[ 2023-11-24T06:07:58.355 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:07:58.355 INFO:teuthology.orchestra.run.smithi091.stdout: "poolnum": 1, 2023-11-24T06:07:58.355 INFO:teuthology.orchestra.run.smithi091.stdout: "poolname": ".mgr" 2023-11-24T06:07:58.355 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:07:58.355 INFO:teuthology.orchestra.run.smithi091.stdout:] 2023-11-24T06:07:58.357 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.354+0000 7fad39812700 1 -- 172.21.15.91:0/3049243926 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad2006aed0 msgr2=0x7fad2006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:58.357 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.354+0000 7fad39812700 1 --2- 172.21.15.91:0/3049243926 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad2006aed0 0x7fad2006d390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fad28015040 tx=0x7fad2800a000 comp rx=0 tx=0).stop 2023-11-24T06:07:58.357 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.354+0000 7fad39812700 1 -- 172.21.15.91:0/3049243926 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad340b5110 msgr2=0x7fad3406fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:58.357 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.354+0000 7fad39812700 1 --2- 172.21.15.91:0/3049243926 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad340b5110 0x7fad3406fa60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fad240119d0 tx=0x7fad24011ce0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.357 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.354+0000 7fad39812700 1 -- 172.21.15.91:0/3049243926 shutdown_connections 2023-11-24T06:07:58.357 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.354+0000 7fad39812700 1 --2- 172.21.15.91:0/3049243926 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad2006aed0 0x7fad2006d390 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.358 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.354+0000 7fad39812700 1 --2- 172.21.15.91:0/3049243926 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad1c003a00 0x7fad34159480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.358 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.354+0000 7fad39812700 1 --2- 172.21.15.91:0/3049243926 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad341f2ba0 0x7fad341f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.358 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.354+0000 7fad39812700 1 --2- 172.21.15.91:0/3049243926 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad340b5110 0x7fad3406fa60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.358 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.354+0000 7fad39812700 1 -- 172.21.15.91:0/3049243926 >> 172.21.15.91:0/3049243926 conn(0x7fad340b3870 msgr2=0x7fad340d0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:58.358 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.354+0000 7fad39812700 1 -- 172.21.15.91:0/3049243926 shutdown_connections 2023-11-24T06:07:58.358 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.354+0000 7fad39812700 1 -- 172.21.15.91:0/3049243926 wait complete. 2023-11-24T06:07:58.369 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T06:07:58.369 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T06:07:58.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.490+0000 7fd467ba4700 1 Processor -- start 2023-11-24T06:07:58.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.490+0000 7fd467ba4700 1 -- start start 2023-11-24T06:07:58.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd467ba4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd46015a440 0x7fd4600b5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:58.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd467ba4700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd4600d0980 0x7fd46015e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:58.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd467ba4700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd46006bd80 con 0x7fd4600ad1f0 2023-11-24T06:07:58.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd467ba4700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4600b6380 con 0x7fd4600d0980 2023-11-24T06:07:58.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd467ba4700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46006b170 con 0x7fd46015a440 2023-11-24T06:07:58.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd466141700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd4600d0980 0x7fd46015e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:58.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd46513f700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fd4600ad1f0 0x7fd4600d0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52524/0 (socket says 172.21.15.91:52524) 2023-11-24T06:07:58.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd46513f700 1 -- 172.21.15.91:0/3382558534 learned_addr learned my addr 172.21.15.91:0/3382558534 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:07:58.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd465940700 1 --2- 172.21.15.91:0/3382558534 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd46015a440 0x7fd4600b5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:58.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd466141700 1 -- 172.21.15.91:0/3382558534 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd46015a440 msgr2=0x7fd4600b5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:58.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd466141700 1 --2- 172.21.15.91:0/3382558534 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd46015a440 0x7fd4600b5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd466141700 1 -- 172.21.15.91:0/3382558534 >> v1:172.21.15.91:6789/0 conn(0x7fd4600ad1f0 legacy=0x7fd4600d0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:58.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd46493e700 1 -- 172.21.15.91:0/3382558534 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029870695 0 0) 0x7fd46006bd80 con 0x7fd4600ad1f0 2023-11-24T06:07:58.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd466141700 1 -- 172.21.15.91:0/3382558534 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46006b450 con 0x7fd4600d0980 2023-11-24T06:07:58.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd465940700 1 --2- 172.21.15.91:0/3382558534 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd46015a440 0x7fd4600b5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:07:58.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd466141700 1 --2- 172.21.15.91:0/3382558534 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4600d0980 0x7fd46015e1c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd458009dc0 tx=0x7fd45800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=800701d797fa1da9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:58.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.491+0000 7fd46493e700 1 -- 172.21.15.91:0/3382558534 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45800e070 con 0x7fd4600d0980 2023-11-24T06:07:58.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.492+0000 7fd46493e700 1 -- 172.21.15.91:0/3382558534 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45800bb90 con 0x7fd4600d0980 2023-11-24T06:07:58.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.492+0000 7fd46493e700 1 -- 172.21.15.91:0/3382558534 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd458012430 con 0x7fd4600d0980 2023-11-24T06:07:58.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.492+0000 7fd467ba4700 1 -- 172.21.15.91:0/3382558534 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4600d0980 msgr2=0x7fd46015e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:58.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.492+0000 7fd467ba4700 1 --2- 172.21.15.91:0/3382558534 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4600d0980 0x7fd46015e1c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd458009dc0 tx=0x7fd45800b330 comp rx=0 tx=0).stop 2023-11-24T06:07:58.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.492+0000 7fd467ba4700 1 -- 172.21.15.91:0/3382558534 shutdown_connections 2023-11-24T06:07:58.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.492+0000 7fd467ba4700 1 --2- 172.21.15.91:0/3382558534 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4600d0980 0x7fd46015e1c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.492+0000 7fd467ba4700 1 --2- 172.21.15.91:0/3382558534 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd46015a440 0x7fd4600b5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.492+0000 7fd467ba4700 1 -- 172.21.15.91:0/3382558534 >> 172.21.15.91:0/3382558534 conn(0x7fd4600b5070 msgr2=0x7fd4600b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:58.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.492+0000 7fd467ba4700 1 -- 172.21.15.91:0/3382558534 shutdown_connections 2023-11-24T06:07:58.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.492+0000 7fd467ba4700 1 -- 172.21.15.91:0/3382558534 wait complete. 2023-11-24T06:07:58.500 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.493+0000 7fd467ba4700 1 Processor -- start 2023-11-24T06:07:58.500 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.493+0000 7fd467ba4700 1 -- start start 2023-11-24T06:07:58.500 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.494+0000 7fd467ba4700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4600ad1f0 0x7fd4600ceba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:58.500 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.494+0000 7fd467ba4700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4600d0980 0x7fd4600ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:58.500 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.494+0000 7fd467ba4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd46015a440 0x7fd4601e2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:58.500 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.494+0000 7fd467ba4700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46014d740 con 0x7fd4600ad1f0 2023-11-24T06:07:58.500 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.494+0000 7fd467ba4700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46014da00 con 0x7fd4600d0980 2023-11-24T06:07:58.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.494+0000 7fd467ba4700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46014dce0 con 0x7fd46015a440 2023-11-24T06:07:58.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.494+0000 7fd465940700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4600ad1f0 0x7fd4600ceba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:58.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.494+0000 7fd465940700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4600ad1f0 0x7fd4600ceba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57596/0 (socket says 172.21.15.91:57596) 2023-11-24T06:07:58.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.494+0000 7fd465940700 1 -- 172.21.15.91:0/1893622471 learned_addr learned my addr 172.21.15.91:0/1893622471 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:58.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.494+0000 7fd46513f700 1 --2- 172.21.15.91:0/1893622471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4600d0980 0x7fd4600ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:58.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.494+0000 7fd466141700 1 --2- 172.21.15.91:0/1893622471 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd46015a440 0x7fd4601e2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:58.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.494+0000 7fd465940700 1 -- 172.21.15.91:0/1893622471 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd46015a440 msgr2=0x7fd4601e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:58.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.494+0000 7fd465940700 1 --2- 172.21.15.91:0/1893622471 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd46015a440 0x7fd4601e2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.494+0000 7fd465940700 1 -- 172.21.15.91:0/1893622471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4600d0980 msgr2=0x7fd4600ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:58.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.494+0000 7fd465940700 1 --2- 172.21.15.91:0/1893622471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4600d0980 0x7fd4600ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:58.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.494+0000 7fd465940700 1 -- 172.21.15.91:0/1893622471 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd458009a70 con 0x7fd4600ad1f0 2023-11-24T06:07:58.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.495+0000 7fd465940700 1 --2- 172.21.15.91:0/1893622471 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4600ad1f0 0x7fd4600ceba0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00c7f0 tx=0x7fd45c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:58.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.495+0000 7fd4567fc700 1 -- 172.21.15.91:0/1893622471 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c011800 con 0x7fd4600ad1f0 2023-11-24T06:07:58.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.495+0000 7fd4567fc700 1 -- 172.21.15.91:0/1893622471 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45c011960 con 0x7fd4600ad1f0 2023-11-24T06:07:58.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.495+0000 7fd4567fc700 1 -- 172.21.15.91:0/1893622471 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c011cf0 con 0x7fd4600ad1f0 2023-11-24T06:07:58.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.495+0000 7fd467ba4700 1 -- 172.21.15.91:0/1893622471 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd46014dfc0 con 0x7fd4600ad1f0 2023-11-24T06:07:58.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.495+0000 7fd467ba4700 1 -- 172.21.15.91:0/1893622471 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4600cd4c0 con 0x7fd4600ad1f0 2023-11-24T06:07:58.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.496+0000 7fd467ba4700 1 -- 172.21.15.91:0/1893622471 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4600b6530 con 0x7fd4600ad1f0 2023-11-24T06:07:58.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.496+0000 7fd4567fc700 1 -- 172.21.15.91:0/1893622471 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd45c018360 con 0x7fd4600ad1f0 2023-11-24T06:07:58.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.497+0000 7fd4567fc700 1 --2- 172.21.15.91:0/1893622471 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd44406aed0 0x7fd44406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:58.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.497+0000 7fd46513f700 1 --2- 172.21.15.91:0/1893622471 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd44406aed0 0x7fd44406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:58.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.497+0000 7fd4567fc700 1 -- 172.21.15.91:0/1893622471 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd45c089180 con 0x7fd4600ad1f0 2023-11-24T06:07:58.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.497+0000 7fd46513f700 1 --2- 172.21.15.91:0/1893622471 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd44406aed0 0x7fd44406d390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd4601e6000 tx=0x7fd450007de0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:58.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.501+0000 7fd4567fc700 1 -- 172.21.15.91:0/1893622471 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd45c0544e0 con 0x7fd4600ad1f0 2023-11-24T06:07:58.695 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:58.692+0000 7fd467ba4700 1 -- 172.21.15.91:0/1893622471 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd460069120 con 0x7fd4600ad1f0 2023-11-24T06:07:59.402 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.399+0000 7fd4567fc700 1 -- 172.21.15.91:0/1893622471 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd45c057b00 con 0x7fd4600ad1f0 2023-11-24T06:07:59.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.403+0000 7fd467ba4700 1 -- 172.21.15.91:0/1893622471 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd44406aed0 msgr2=0x7fd44406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:59.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.403+0000 7fd467ba4700 1 --2- 172.21.15.91:0/1893622471 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd44406aed0 0x7fd44406d390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd4601e6000 tx=0x7fd450007de0 comp rx=0 tx=0).stop 2023-11-24T06:07:59.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.403+0000 7fd467ba4700 1 -- 172.21.15.91:0/1893622471 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4600ad1f0 msgr2=0x7fd4600ceba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:59.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.403+0000 7fd467ba4700 1 --2- 172.21.15.91:0/1893622471 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4600ad1f0 0x7fd4600ceba0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00c7f0 tx=0x7fd45c00cb00 comp rx=0 tx=0).stop 2023-11-24T06:07:59.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.404+0000 7fd467ba4700 1 -- 172.21.15.91:0/1893622471 shutdown_connections 2023-11-24T06:07:59.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.404+0000 7fd467ba4700 1 --2- 172.21.15.91:0/1893622471 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd44406aed0 0x7fd44406d390 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:59.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.404+0000 7fd467ba4700 1 --2- 172.21.15.91:0/1893622471 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd46015a440 0x7fd4601e2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:59.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.404+0000 7fd467ba4700 1 --2- 172.21.15.91:0/1893622471 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4600ad1f0 0x7fd4600ceba0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:59.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.404+0000 7fd467ba4700 1 --2- 172.21.15.91:0/1893622471 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4600d0980 0x7fd4600ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:59.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.404+0000 7fd467ba4700 1 -- 172.21.15.91:0/1893622471 >> 172.21.15.91:0/1893622471 conn(0x7fd4600b5070 msgr2=0x7fd46015d470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:59.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.404+0000 7fd467ba4700 1 -- 172.21.15.91:0/1893622471 shutdown_connections 2023-11-24T06:07:59.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.404+0000 7fd467ba4700 1 -- 172.21.15.91:0/1893622471 wait complete. 2023-11-24T06:07:59.408 INFO:teuthology.orchestra.run.smithi091.stderr:pool 'cephfs_metadata' created 2023-11-24T06:07:59.424 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T06:07:59.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.556+0000 7f48b526c700 1 Processor -- start 2023-11-24T06:07:59.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.556+0000 7f48b526c700 1 -- start start 2023-11-24T06:07:59.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48b526c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f48b0161ad0 0x7f48b00d2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:59.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48b526c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48b00b47e0 0x7f48b00cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:59.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48b526c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b00d29b0 con 0x7f48b01616f0 2023-11-24T06:07:59.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48b526c700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b00cffc0 con 0x7f48b0161ad0 2023-11-24T06:07:59.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48b526c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00d2c70 con 0x7f48b00b47e0 2023-11-24T06:07:59.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48ae7fc700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f48b0161ad0 0x7f48b00d2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:59.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48ae7fc700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f48b0161ad0 0x7f48b00d2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57606/0 (socket says 172.21.15.91:57606) 2023-11-24T06:07:59.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48ae7fc700 1 -- 172.21.15.91:0/4072252081 learned_addr learned my addr 172.21.15.91:0/4072252081 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:59.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48af7fe700 1 --2- 172.21.15.91:0/4072252081 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48b00b47e0 0x7f48b00cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:59.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48ae7fc700 1 -- 172.21.15.91:0/4072252081 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48b00b47e0 msgr2=0x7f48b00cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:59.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48ae7fc700 1 --2- 172.21.15.91:0/4072252081 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48b00b47e0 0x7f48b00cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:59.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48ae7fc700 1 -- 172.21.15.91:0/4072252081 >> v1:172.21.15.94:6789/0 conn(0x7f48b01616f0 legacy=0x7f48b00d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:07:59.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48ae7fc700 1 -- 172.21.15.91:0/4072252081 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b00d0f20 con 0x7f48b0161ad0 2023-11-24T06:07:59.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48af7fe700 1 --2- 172.21.15.91:0/4072252081 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48b00b47e0 0x7f48b00cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:07:59.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48ae7fc700 1 --2- 172.21.15.91:0/4072252081 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f48b0161ad0 0x7f48b00d2470 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f48a40098f0 tx=0x7f48a4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e971fbb94ec63011 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:59.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48adffb700 1 -- 172.21.15.91:0/4072252081 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a4004230 con 0x7f48b0161ad0 2023-11-24T06:07:59.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48adffb700 1 -- 172.21.15.91:0/4072252081 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a4004390 con 0x7f48b0161ad0 2023-11-24T06:07:59.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.557+0000 7f48adffb700 1 -- 172.21.15.91:0/4072252081 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a40036a0 con 0x7f48b0161ad0 2023-11-24T06:07:59.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.558+0000 7f48b526c700 1 -- 172.21.15.91:0/4072252081 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f48b0161ad0 msgr2=0x7f48b00d2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:59.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.558+0000 7f48b526c700 1 --2- 172.21.15.91:0/4072252081 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f48b0161ad0 0x7f48b00d2470 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f48a40098f0 tx=0x7f48a4009c00 comp rx=0 tx=0).stop 2023-11-24T06:07:59.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.558+0000 7f48b526c700 1 -- 172.21.15.91:0/4072252081 shutdown_connections 2023-11-24T06:07:59.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.558+0000 7f48b526c700 1 --2- 172.21.15.91:0/4072252081 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48b00b47e0 0x7f48b00cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:59.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.558+0000 7f48b526c700 1 --2- 172.21.15.91:0/4072252081 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f48b0161ad0 0x7f48b00d2470 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:59.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.558+0000 7f48b526c700 1 -- 172.21.15.91:0/4072252081 >> 172.21.15.91:0/4072252081 conn(0x7f48b00d3e60 msgr2=0x7f48b00d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:07:59.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.558+0000 7f48b526c700 1 -- 172.21.15.91:0/4072252081 shutdown_connections 2023-11-24T06:07:59.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.558+0000 7f48b526c700 1 -- 172.21.15.91:0/4072252081 wait complete. 2023-11-24T06:07:59.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.559+0000 7f48b526c700 1 Processor -- start 2023-11-24T06:07:59.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.559+0000 7f48b526c700 1 -- start start 2023-11-24T06:07:59.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48b526c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48b00b47e0 0x7f48b00c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:59.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48b526c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f48b01616f0 0x7f48b00cbb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:59.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48b526c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48b0161ad0 0x7f48b00cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:59.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48b526c700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00b7cc0 con 0x7f48b01616f0 2023-11-24T06:07:59.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48b526c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00b7fa0 con 0x7f48b0161ad0 2023-11-24T06:07:59.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48b526c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00b8280 con 0x7f48b00b47e0 2023-11-24T06:07:59.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48ae7fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f48b01616f0 0x7f48b00cbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:59.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48ae7fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f48b01616f0 0x7f48b00cbb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57612/0 (socket says 172.21.15.91:57612) 2023-11-24T06:07:59.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48ae7fc700 1 -- 172.21.15.91:0/3966660673 learned_addr learned my addr 172.21.15.91:0/3966660673 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:07:59.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48af7fe700 1 --2- 172.21.15.91:0/3966660673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48b0161ad0 0x7f48b00cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:59.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48aeffd700 1 --2- 172.21.15.91:0/3966660673 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48b00b47e0 0x7f48b00c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:59.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48ae7fc700 1 -- 172.21.15.91:0/3966660673 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48b00b47e0 msgr2=0x7f48b00c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:59.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48ae7fc700 1 --2- 172.21.15.91:0/3966660673 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48b00b47e0 0x7f48b00c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:59.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48ae7fc700 1 -- 172.21.15.91:0/3966660673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48b0161ad0 msgr2=0x7f48b00cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:07:59.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48ae7fc700 1 --2- 172.21.15.91:0/3966660673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48b0161ad0 0x7f48b00cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:07:59.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48ae7fc700 1 -- 172.21.15.91:0/3966660673 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a4009550 con 0x7f48b01616f0 2023-11-24T06:07:59.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.560+0000 7f48ae7fc700 1 --2- 172.21.15.91:0/3966660673 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f48b01616f0 0x7f48b00cbb40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f48a4009130 tx=0x7f48a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:59.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.561+0000 7f48977fe700 1 -- 172.21.15.91:0/3966660673 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a400b660 con 0x7f48b01616f0 2023-11-24T06:07:59.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.561+0000 7f48977fe700 1 -- 172.21.15.91:0/3966660673 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a400b7c0 con 0x7f48b01616f0 2023-11-24T06:07:59.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.561+0000 7f48977fe700 1 -- 172.21.15.91:0/3966660673 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a400bb70 con 0x7f48b01616f0 2023-11-24T06:07:59.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.561+0000 7f48b526c700 1 -- 172.21.15.91:0/3966660673 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b00b8560 con 0x7f48b01616f0 2023-11-24T06:07:59.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.561+0000 7f48b526c700 1 -- 172.21.15.91:0/3966660673 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b00c7de0 con 0x7f48b01616f0 2023-11-24T06:07:59.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.562+0000 7f48b526c700 1 -- 172.21.15.91:0/3966660673 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b0068480 con 0x7f48b01616f0 2023-11-24T06:07:59.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.562+0000 7f48977fe700 1 -- 172.21.15.91:0/3966660673 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f48a400bcd0 con 0x7f48b01616f0 2023-11-24T06:07:59.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.562+0000 7f48977fe700 1 --2- 172.21.15.91:0/3966660673 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f489c06aed0 0x7f489c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:07:59.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.562+0000 7f48977fe700 1 -- 172.21.15.91:0/3966660673 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f48a408ac00 con 0x7f48b01616f0 2023-11-24T06:07:59.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.562+0000 7f48aeffd700 1 --2- 172.21.15.91:0/3966660673 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f489c06aed0 0x7f489c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:07:59.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.563+0000 7f48aeffd700 1 --2- 172.21.15.91:0/3966660673 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f489c06aed0 0x7f489c06d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f48a0007fd0 tx=0x7f48a0007ec0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:07:59.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.567+0000 7f48977fe700 1 -- 172.21.15.91:0/3966660673 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f48a4091080 con 0x7f48b01616f0 2023-11-24T06:07:59.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:07:59.756+0000 7f48b526c700 1 -- 172.21.15.91:0/3966660673 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f48b0075480 con 0x7f48b01616f0 2023-11-24T06:08:00.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.402+0000 7f48977fe700 1 -- 172.21.15.91:0/3966660673 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7f48a4055f20 con 0x7f48b01616f0 2023-11-24T06:08:00.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.405+0000 7f48b526c700 1 -- 172.21.15.91:0/3966660673 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f489c06aed0 msgr2=0x7f489c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:00.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.405+0000 7f48b526c700 1 --2- 172.21.15.91:0/3966660673 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f489c06aed0 0x7f489c06d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f48a0007fd0 tx=0x7f48a0007ec0 comp rx=0 tx=0).stop 2023-11-24T06:08:00.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.405+0000 7f48b526c700 1 -- 172.21.15.91:0/3966660673 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f48b01616f0 msgr2=0x7f48b00cbb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:00.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.405+0000 7f48b526c700 1 --2- 172.21.15.91:0/3966660673 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f48b01616f0 0x7f48b00cbb40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f48a4009130 tx=0x7f48a4004750 comp rx=0 tx=0).stop 2023-11-24T06:08:00.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.405+0000 7f48b526c700 1 -- 172.21.15.91:0/3966660673 shutdown_connections 2023-11-24T06:08:00.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.405+0000 7f48b526c700 1 --2- 172.21.15.91:0/3966660673 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f489c06aed0 0x7f489c06d390 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:00.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.405+0000 7f48b526c700 1 --2- 172.21.15.91:0/3966660673 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f48b0161ad0 0x7f48b00cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:00.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.405+0000 7f48b526c700 1 --2- 172.21.15.91:0/3966660673 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f48b00b47e0 0x7f48b00c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:00.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.405+0000 7f48b526c700 1 --2- 172.21.15.91:0/3966660673 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f48b01616f0 0x7f48b00cbb40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:00.409 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.405+0000 7f48b526c700 1 -- 172.21.15.91:0/3966660673 >> 172.21.15.91:0/3966660673 conn(0x7f48b00d3e60 msgr2=0x7f48b00cea70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:00.409 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.405+0000 7f48b526c700 1 -- 172.21.15.91:0/3966660673 shutdown_connections 2023-11-24T06:08:00.409 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.405+0000 7f48b526c700 1 -- 172.21.15.91:0/3966660673 wait complete. 2023-11-24T06:08:00.420 INFO:teuthology.orchestra.run.smithi091.stderr:pool 'cephfs_data' created 2023-11-24T06:08:00.430 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T06:08:00.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.550+0000 7f9ac3e78700 1 Processor -- start 2023-11-24T06:08:00.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.551+0000 7f9ac3e78700 1 -- start start 2023-11-24T06:08:00.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.551+0000 7f9ac3e78700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9abc15bde0 0x7f9abc15c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:00.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.551+0000 7f9ac3e78700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f9abc0cf2b0 0x7f9abc0d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:00.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.551+0000 7f9ac3e78700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9abc0ce5c0 0x7f9abc0cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:00.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.551+0000 7f9ac3e78700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9abc0cefb0 con 0x7f9abc0cf2b0 2023-11-24T06:08:00.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.551+0000 7f9ac3e78700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9abc0d0a90 con 0x7f9abc15bde0 2023-11-24T06:08:00.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.551+0000 7f9ac3e78700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc06ee10 con 0x7f9abc0ce5c0 2023-11-24T06:08:00.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.551+0000 7f9ac1413700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f9abc0cf2b0 0x7f9abc0d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:00.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.551+0000 7f9ac1413700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f9abc0cf2b0 0x7f9abc0d01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57618/0 (socket says 172.21.15.91:57618) 2023-11-24T06:08:00.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.552+0000 7f9ac1413700 1 -- 172.21.15.91:0/85459035 learned_addr learned my addr 172.21.15.91:0/85459035 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:00.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.552+0000 7f9ac1c14700 1 --2- 172.21.15.91:0/85459035 >> v2:172.21.15.94:3300/0 conn(0x7f9abc15bde0 0x7f9abc15c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:00.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.552+0000 7f9ac2415700 1 --2- 172.21.15.91:0/85459035 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9abc0ce5c0 0x7f9abc0cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:00.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.552+0000 7f9ac1413700 1 -- 172.21.15.91:0/85459035 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9abc0ce5c0 msgr2=0x7f9abc0cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:00.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.552+0000 7f9ac1413700 1 --2- 172.21.15.91:0/85459035 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9abc0ce5c0 0x7f9abc0cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:00.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.552+0000 7f9ac1413700 1 -- 172.21.15.91:0/85459035 >> v2:172.21.15.94:3300/0 conn(0x7f9abc15bde0 msgr2=0x7f9abc15c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:00.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.552+0000 7f9ac1413700 1 --2- 172.21.15.91:0/85459035 >> v2:172.21.15.94:3300/0 conn(0x7f9abc15bde0 0x7f9abc15c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:00.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.552+0000 7f9ac1413700 1 -- 172.21.15.91:0/85459035 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc06f0f0 con 0x7f9abc0cf2b0 2023-11-24T06:08:00.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.552+0000 7f9ac1413700 1 --2- 172.21.15.91:0/85459035 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9abc0cf2b0 0x7f9abc0d01b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9abc072370 tx=0x7f9ab8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d5a29cd9f75e5e7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:00.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.552+0000 7f9ac0c12700 1 -- 172.21.15.91:0/85459035 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab800b870 con 0x7f9abc0cf2b0 2023-11-24T06:08:00.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.552+0000 7f9ac0c12700 1 -- 172.21.15.91:0/85459035 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab800b9d0 con 0x7f9abc0cf2b0 2023-11-24T06:08:00.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.552+0000 7f9ac0c12700 1 -- 172.21.15.91:0/85459035 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab800bd60 con 0x7f9abc0cf2b0 2023-11-24T06:08:00.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.553+0000 7f9ac3e78700 1 -- 172.21.15.91:0/85459035 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9abc0cf2b0 msgr2=0x7f9abc0d01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:00.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.553+0000 7f9ac3e78700 1 --2- 172.21.15.91:0/85459035 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9abc0cf2b0 0x7f9abc0d01b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9abc072370 tx=0x7f9ab8009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:00.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.553+0000 7f9ac3e78700 1 -- 172.21.15.91:0/85459035 shutdown_connections 2023-11-24T06:08:00.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.553+0000 7f9ac3e78700 1 --2- 172.21.15.91:0/85459035 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9abc0ce5c0 0x7f9abc0cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:00.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.553+0000 7f9ac3e78700 1 --2- 172.21.15.91:0/85459035 >> v2:172.21.15.94:3300/0 conn(0x7f9abc15bde0 0x7f9abc15c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:00.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.553+0000 7f9ac3e78700 1 --2- 172.21.15.91:0/85459035 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9abc0cf2b0 0x7f9abc0d01b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:00.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.553+0000 7f9ac3e78700 1 -- 172.21.15.91:0/85459035 >> 172.21.15.91:0/85459035 conn(0x7f9abc0b3720 msgr2=0x7f9abc0b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:00.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.553+0000 7f9ac3e78700 1 -- 172.21.15.91:0/85459035 shutdown_connections 2023-11-24T06:08:00.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.553+0000 7f9ac3e78700 1 -- 172.21.15.91:0/85459035 wait complete. 2023-11-24T06:08:00.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.554+0000 7f9ac3e78700 1 Processor -- start 2023-11-24T06:08:00.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.554+0000 7f9ac3e78700 1 -- start start 2023-11-24T06:08:00.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac3e78700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9abc0ce5c0 0x7f9abc0afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:00.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac3e78700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9abc0cf2b0 0x7f9abc0c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:00.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac3e78700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9abc15bde0 0x7f9abc0c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:00.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac3e78700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc150320 con 0x7f9abc0cf2b0 2023-11-24T06:08:00.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac3e78700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc150530 con 0x7f9abc0ce5c0 2023-11-24T06:08:00.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac3e78700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9abc150810 con 0x7f9abc15bde0 2023-11-24T06:08:00.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac1413700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9abc0cf2b0 0x7f9abc0c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:00.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac1413700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9abc0cf2b0 0x7f9abc0c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57626/0 (socket says 172.21.15.91:57626) 2023-11-24T06:08:00.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac1413700 1 -- 172.21.15.91:0/2500617966 learned_addr learned my addr 172.21.15.91:0/2500617966 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:00.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac1c14700 1 --2- 172.21.15.91:0/2500617966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9abc0ce5c0 0x7f9abc0afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:00.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac2415700 1 --2- 172.21.15.91:0/2500617966 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9abc15bde0 0x7f9abc0c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:00.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac1413700 1 -- 172.21.15.91:0/2500617966 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9abc15bde0 msgr2=0x7f9abc0c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:00.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac1413700 1 --2- 172.21.15.91:0/2500617966 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9abc15bde0 0x7f9abc0c7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:00.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac1413700 1 -- 172.21.15.91:0/2500617966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9abc0ce5c0 msgr2=0x7f9abc0afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:00.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac1413700 1 --2- 172.21.15.91:0/2500617966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9abc0ce5c0 0x7f9abc0afe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:00.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac1413700 1 -- 172.21.15.91:0/2500617966 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab8009550 con 0x7f9abc0cf2b0 2023-11-24T06:08:00.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac2415700 1 --2- 172.21.15.91:0/2500617966 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9abc15bde0 0x7f9abc0c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:00.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac1413700 1 --2- 172.21.15.91:0/2500617966 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9abc0cf2b0 0x7f9abc0c77b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8000c00 tx=0x7f9ab8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:00.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9aae7fc700 1 -- 172.21.15.91:0/2500617966 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8004540 con 0x7f9abc0cf2b0 2023-11-24T06:08:00.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9aae7fc700 1 -- 172.21.15.91:0/2500617966 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab80036a0 con 0x7f9abc0cf2b0 2023-11-24T06:08:00.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac3e78700 1 -- 172.21.15.91:0/2500617966 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9abc150af0 con 0x7f9abc0cf2b0 2023-11-24T06:08:00.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9aae7fc700 1 -- 172.21.15.91:0/2500617966 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8003a50 con 0x7f9abc0cf2b0 2023-11-24T06:08:00.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.555+0000 7f9ac3e78700 1 -- 172.21.15.91:0/2500617966 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9abc0c8230 con 0x7f9abc0cf2b0 2023-11-24T06:08:00.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.556+0000 7f9ac3e78700 1 -- 172.21.15.91:0/2500617966 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9abc0ce390 con 0x7f9abc0cf2b0 2023-11-24T06:08:00.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.557+0000 7f9aae7fc700 1 -- 172.21.15.91:0/2500617966 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9ab8021070 con 0x7f9abc0cf2b0 2023-11-24T06:08:00.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.557+0000 7f9aae7fc700 1 --2- 172.21.15.91:0/2500617966 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9aa006aed0 0x7f9aa006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:00.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.557+0000 7f9aae7fc700 1 -- 172.21.15.91:0/2500617966 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f9ab808acc0 con 0x7f9abc0cf2b0 2023-11-24T06:08:00.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.557+0000 7f9ac1c14700 1 --2- 172.21.15.91:0/2500617966 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9aa006aed0 0x7f9aa006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:00.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.558+0000 7f9ac1c14700 1 --2- 172.21.15.91:0/2500617966 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9aa006aed0 0x7f9aa006d390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0006b20 tx=0x7f9ab0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:00.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.562+0000 7f9aae7fc700 1 -- 172.21.15.91:0/2500617966 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9ab8055f80 con 0x7f9abc0cf2b0 2023-11-24T06:08:00.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:00.797+0000 7f9ac3e78700 1 -- 172.21.15.91:0/2500617966 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f9abc075480 con 0x7f9abc0cf2b0 2023-11-24T06:08:01.305 DEBUG:teuthology.orchestra.run.smithi091:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:08:01.307 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:08:01.730 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.727+0000 7f9aae7fc700 1 -- 172.21.15.91:0/2500617966 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f9ab8015020 con 0x7f9abc0cf2b0 2023-11-24T06:08:01.732 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.729+0000 7f9ac3e78700 1 -- 172.21.15.91:0/2500617966 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9aa006aed0 msgr2=0x7f9aa006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:01.732 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.729+0000 7f9ac3e78700 1 --2- 172.21.15.91:0/2500617966 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9aa006aed0 0x7f9aa006d390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0006b20 tx=0x7f9ab0008040 comp rx=0 tx=0).stop 2023-11-24T06:08:01.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.729+0000 7f9ac3e78700 1 -- 172.21.15.91:0/2500617966 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9abc0cf2b0 msgr2=0x7f9abc0c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:01.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.729+0000 7f9ac3e78700 1 --2- 172.21.15.91:0/2500617966 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9abc0cf2b0 0x7f9abc0c77b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8000c00 tx=0x7f9ab8004750 comp rx=0 tx=0).stop 2023-11-24T06:08:01.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.729+0000 7f9ac3e78700 1 -- 172.21.15.91:0/2500617966 shutdown_connections 2023-11-24T06:08:01.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.729+0000 7f9ac3e78700 1 --2- 172.21.15.91:0/2500617966 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9aa006aed0 0x7f9aa006d390 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:01.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.729+0000 7f9ac3e78700 1 --2- 172.21.15.91:0/2500617966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9abc0ce5c0 0x7f9abc0afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:01.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.730+0000 7f9ac3e78700 1 --2- 172.21.15.91:0/2500617966 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9abc15bde0 0x7f9abc0c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:01.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.730+0000 7f9ac3e78700 1 --2- 172.21.15.91:0/2500617966 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9abc0cf2b0 0x7f9abc0c77b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:01.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.730+0000 7f9ac3e78700 1 -- 172.21.15.91:0/2500617966 >> 172.21.15.91:0/2500617966 conn(0x7f9abc0b3720 msgr2=0x7f9abc15a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:01.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.730+0000 7f9ac3e78700 1 -- 172.21.15.91:0/2500617966 shutdown_connections 2023-11-24T06:08:01.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.730+0000 7f9ac3e78700 1 -- 172.21.15.91:0/2500617966 wait complete. 2023-11-24T06:08:01.736 INFO:teuthology.orchestra.run.smithi091.stderr:new fs with metadata pool 2 and data pool 3 2023-11-24T06:08:01.749 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:01.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.866+0000 7f6dcb271700 1 Processor -- start 2023-11-24T06:08:01.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.866+0000 7f6dcb271700 1 -- start start 2023-11-24T06:08:01.872 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.867+0000 7f6dcb271700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f6dc4143ae0 0x7f6dc40b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:01.872 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.867+0000 7f6dcb271700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6dc41440f0 0x7f6dc4071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:01.872 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.867+0000 7f6dcb271700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc4143ec0 con 0x7f6dc40720f0 2023-11-24T06:08:01.872 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.867+0000 7f6dcb271700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dc40b40a0 con 0x7f6dc4143ae0 2023-11-24T06:08:01.872 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.867+0000 7f6dcb271700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc4072580 con 0x7f6dc41440f0 2023-11-24T06:08:01.872 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.867+0000 7f6dc900d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f6dc4143ae0 0x7f6dc40b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:01.872 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.867+0000 7f6dc900d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f6dc4143ae0 0x7f6dc40b51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57646/0 (socket says 172.21.15.91:57646) 2023-11-24T06:08:01.872 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.867+0000 7f6dc900d700 1 -- 172.21.15.91:0/2035573609 learned_addr learned my addr 172.21.15.91:0/2035573609 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:01.872 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.867+0000 7f6dc880c700 1 --2- 172.21.15.91:0/2035573609 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6dc41440f0 0x7f6dc4071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:01.872 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.867+0000 7f6dc900d700 1 -- 172.21.15.91:0/2035573609 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6dc41440f0 msgr2=0x7f6dc4071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:01.872 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.867+0000 7f6dc900d700 1 --2- 172.21.15.91:0/2035573609 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6dc41440f0 0x7f6dc4071bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:01.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.867+0000 7f6dc900d700 1 -- 172.21.15.91:0/2035573609 >> v1:172.21.15.94:6789/0 conn(0x7f6dc40720f0 legacy=0x7f6dc415e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:01.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.867+0000 7f6dc900d700 1 -- 172.21.15.91:0/2035573609 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc40b3ae0 con 0x7f6dc4143ae0 2023-11-24T06:08:01.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.867+0000 7f6dc880c700 1 --2- 172.21.15.91:0/2035573609 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6dc41440f0 0x7f6dc4071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:01.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.867+0000 7f6dc900d700 1 --2- 172.21.15.91:0/2035573609 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6dc4143ae0 0x7f6dc40b51b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6dc00098f0 tx=0x7f6dc0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=609cfb74c20ef885 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:01.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.867+0000 7f6dbbfff700 1 -- 172.21.15.91:0/2035573609 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc000b8a0 con 0x7f6dc4143ae0 2023-11-24T06:08:01.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.868+0000 7f6dbbfff700 1 -- 172.21.15.91:0/2035573609 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc000ba00 con 0x7f6dc4143ae0 2023-11-24T06:08:01.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.868+0000 7f6dbbfff700 1 -- 172.21.15.91:0/2035573609 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc000bd90 con 0x7f6dc4143ae0 2023-11-24T06:08:01.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.868+0000 7f6dcb271700 1 -- 172.21.15.91:0/2035573609 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6dc4143ae0 msgr2=0x7f6dc40b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:01.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.868+0000 7f6dcb271700 1 --2- 172.21.15.91:0/2035573609 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6dc4143ae0 0x7f6dc40b51b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6dc00098f0 tx=0x7f6dc0009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:01.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.868+0000 7f6dcb271700 1 -- 172.21.15.91:0/2035573609 shutdown_connections 2023-11-24T06:08:01.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.868+0000 7f6dcb271700 1 --2- 172.21.15.91:0/2035573609 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6dc4143ae0 0x7f6dc40b51b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:01.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.868+0000 7f6dcb271700 1 --2- 172.21.15.91:0/2035573609 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6dc41440f0 0x7f6dc4071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:01.874 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.868+0000 7f6dcb271700 1 -- 172.21.15.91:0/2035573609 >> 172.21.15.91:0/2035573609 conn(0x7f6dc40b2740 msgr2=0x7f6dc40b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:01.874 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.868+0000 7f6dcb271700 1 -- 172.21.15.91:0/2035573609 shutdown_connections 2023-11-24T06:08:01.874 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.868+0000 7f6dcb271700 1 -- 172.21.15.91:0/2035573609 wait complete. 2023-11-24T06:08:01.876 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.870+0000 7f6dcb271700 1 Processor -- start 2023-11-24T06:08:01.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.870+0000 7f6dcb271700 1 -- start start 2023-11-24T06:08:01.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dcb271700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6dc40720f0 0x7f6dc413e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:01.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dcb271700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6dc4143ae0 0x7f6dc414f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:01.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dcb271700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dc41440f0 0x7f6dc40ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:01.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dcb271700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc4150e70 con 0x7f6dc4143ae0 2023-11-24T06:08:01.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dcb271700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc414fdd0 con 0x7f6dc41440f0 2023-11-24T06:08:01.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dcb271700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc41500b0 con 0x7f6dc40720f0 2023-11-24T06:08:01.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dc880c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6dc4143ae0 0x7f6dc414f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:01.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dc880c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6dc4143ae0 0x7f6dc414f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57654/0 (socket says 172.21.15.91:57654) 2023-11-24T06:08:01.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dc880c700 1 -- 172.21.15.91:0/1985447989 learned_addr learned my addr 172.21.15.91:0/1985447989 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:01.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dc900d700 1 --2- 172.21.15.91:0/1985447989 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6dc40720f0 0x7f6dc413e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:01.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dc980e700 1 --2- 172.21.15.91:0/1985447989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dc41440f0 0x7f6dc40ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:01.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dc880c700 1 -- 172.21.15.91:0/1985447989 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6dc40720f0 msgr2=0x7f6dc413e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:01.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dc880c700 1 --2- 172.21.15.91:0/1985447989 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6dc40720f0 0x7f6dc413e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:01.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dc880c700 1 -- 172.21.15.91:0/1985447989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dc41440f0 msgr2=0x7f6dc40ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:01.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dc880c700 1 --2- 172.21.15.91:0/1985447989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dc41440f0 0x7f6dc40ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:01.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dc880c700 1 -- 172.21.15.91:0/1985447989 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc0009550 con 0x7f6dc4143ae0 2023-11-24T06:08:01.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dc900d700 1 --2- 172.21.15.91:0/1985447989 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6dc40720f0 0x7f6dc413e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:01.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dc980e700 1 --2- 172.21.15.91:0/1985447989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dc41440f0 0x7f6dc40ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:01.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dc880c700 1 --2- 172.21.15.91:0/1985447989 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6dc4143ae0 0x7f6dc414f3f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6db400c750 tx=0x7f6db400cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:01.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6db9ffb700 1 -- 172.21.15.91:0/1985447989 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db4011800 con 0x7f6dc4143ae0 2023-11-24T06:08:01.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6db9ffb700 1 -- 172.21.15.91:0/1985447989 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db4011960 con 0x7f6dc4143ae0 2023-11-24T06:08:01.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6dcb271700 1 -- 172.21.15.91:0/1985447989 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc4150390 con 0x7f6dc4143ae0 2023-11-24T06:08:01.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.871+0000 7f6db9ffb700 1 -- 172.21.15.91:0/1985447989 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db4011cd0 con 0x7f6dc4143ae0 2023-11-24T06:08:01.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.872+0000 7f6dcb271700 1 -- 172.21.15.91:0/1985447989 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc4150950 con 0x7f6dc4143ae0 2023-11-24T06:08:01.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.872+0000 7f6dcb271700 1 -- 172.21.15.91:0/1985447989 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc415e520 con 0x7f6dc4143ae0 2023-11-24T06:08:01.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.873+0000 7f6db9ffb700 1 -- 172.21.15.91:0/1985447989 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6db4021350 con 0x7f6dc4143ae0 2023-11-24T06:08:01.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.873+0000 7f6db9ffb700 1 --2- 172.21.15.91:0/1985447989 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6da806aed0 0x7f6da806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:01.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.873+0000 7f6db9ffb700 1 -- 172.21.15.91:0/1985447989 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6db40894d0 con 0x7f6dc4143ae0 2023-11-24T06:08:01.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.873+0000 7f6dc900d700 1 --2- 172.21.15.91:0/1985447989 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6da806aed0 0x7f6da806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:01.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.873+0000 7f6dc900d700 1 --2- 172.21.15.91:0/1985447989 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6da806aed0 0x7f6da806d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4168cc0 tx=0x7f6dc0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:01.880 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:01.877+0000 7f6db9ffb700 1 -- 172.21.15.91:0/1985447989 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6db4054610 con 0x7f6dc4143ae0 2023-11-24T06:08:02.073 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.070+0000 7f6dcb271700 1 -- 172.21.15.91:0/1985447989 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6dc4144830 con 0x7f6dc4143ae0 2023-11-24T06:08:02.074 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.071+0000 7f6db9ffb700 1 -- 172.21.15.91:0/1985447989 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16434 (secure 0 0 0) 0x7f6db4057c30 con 0x7f6dc4143ae0 2023-11-24T06:08:02.075 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:02.076 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":19,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:00.798347+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:02.076 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.073+0000 7f6dcb271700 1 -- 172.21.15.91:0/1985447989 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6da806aed0 msgr2=0x7f6da806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:02.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.073+0000 7f6dcb271700 1 --2- 172.21.15.91:0/1985447989 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6da806aed0 0x7f6da806d390 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4168cc0 tx=0x7f6dc0008030 comp rx=0 tx=0).stop 2023-11-24T06:08:02.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.073+0000 7f6dcb271700 1 -- 172.21.15.91:0/1985447989 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6dc4143ae0 msgr2=0x7f6dc414f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:02.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.073+0000 7f6dcb271700 1 --2- 172.21.15.91:0/1985447989 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6dc4143ae0 0x7f6dc414f3f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6db400c750 tx=0x7f6db400cb10 comp rx=0 tx=0).stop 2023-11-24T06:08:02.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.073+0000 7f6dcb271700 1 -- 172.21.15.91:0/1985447989 shutdown_connections 2023-11-24T06:08:02.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.073+0000 7f6dcb271700 1 --2- 172.21.15.91:0/1985447989 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6da806aed0 0x7f6da806d390 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.073+0000 7f6dcb271700 1 --2- 172.21.15.91:0/1985447989 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6dc41440f0 0x7f6dc40ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.073+0000 7f6dcb271700 1 --2- 172.21.15.91:0/1985447989 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6dc40720f0 0x7f6dc413e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.073+0000 7f6dcb271700 1 --2- 172.21.15.91:0/1985447989 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6dc4143ae0 0x7f6dc414f3f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.073+0000 7f6dcb271700 1 -- 172.21.15.91:0/1985447989 >> 172.21.15.91:0/1985447989 conn(0x7f6dc40b2740 msgr2=0x7f6dc406e510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:02.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.073+0000 7f6dcb271700 1 -- 172.21.15.91:0/1985447989 shutdown_connections 2023-11-24T06:08:02.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.073+0000 7f6dcb271700 1 -- 172.21.15.91:0/1985447989 wait complete. 2023-11-24T06:08:02.088 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:02.208 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.205+0000 7f73b4525700 1 Processor -- start 2023-11-24T06:08:02.208 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.205+0000 7f73b4525700 1 -- start start 2023-11-24T06:08:02.209 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.206+0000 7f73b4525700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f73ac0b4150 0x7f73ac06de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:02.209 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.206+0000 7f73b4525700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73ac0b5ab0 0x7f73ac06e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:02.209 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.206+0000 7f73b4525700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ac071f50 con 0x7f73ac06b620 2023-11-24T06:08:02.209 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.206+0000 7f73b4525700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73ac072230 con 0x7f73ac0b4150 2023-11-24T06:08:02.209 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.206+0000 7f73b4525700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac072510 con 0x7f73ac0b5ab0 2023-11-24T06:08:02.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.207+0000 7f73b2ac2700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73ac0b5ab0 0x7f73ac06e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:02.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.207+0000 7f73b1ac0700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f73ac0b4150 0x7f73ac06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:02.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.207+0000 7f73b2ac2700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73ac0b5ab0 0x7f73ac06e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:41096/0 (socket says 172.21.15.91:41096) 2023-11-24T06:08:02.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.207+0000 7f73b2ac2700 1 -- 172.21.15.91:0/515177656 learned_addr learned my addr 172.21.15.91:0/515177656 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:02.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.207+0000 7f73b1ac0700 1 -- 172.21.15.91:0/515177656 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73ac0b5ab0 msgr2=0x7f73ac06e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:02.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.207+0000 7f73b1ac0700 1 --2- 172.21.15.91:0/515177656 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73ac0b5ab0 0x7f73ac06e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.207+0000 7f73b1ac0700 1 -- 172.21.15.91:0/515177656 >> v1:172.21.15.94:6789/0 conn(0x7f73ac06b620 legacy=0x7f73ac0b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:02.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.207+0000 7f73b1ac0700 1 -- 172.21.15.91:0/515177656 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac0727f0 con 0x7f73ac0b4150 2023-11-24T06:08:02.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.207+0000 7f73b2ac2700 1 --2- 172.21.15.91:0/515177656 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73ac0b5ab0 0x7f73ac06e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:08:02.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.207+0000 7f73b1ac0700 1 --2- 172.21.15.91:0/515177656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73ac0b4150 0x7f73ac06de50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009840 tx=0x7f73a8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=414cf888436eb8e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:02.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.207+0000 7f73b12bf700 1 -- 172.21.15.91:0/515177656 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a8011830 con 0x7f73ac0b4150 2023-11-24T06:08:02.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.207+0000 7f73b12bf700 1 -- 172.21.15.91:0/515177656 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a8011990 con 0x7f73ac0b4150 2023-11-24T06:08:02.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.207+0000 7f73b12bf700 1 -- 172.21.15.91:0/515177656 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a8011d20 con 0x7f73ac0b4150 2023-11-24T06:08:02.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.208+0000 7f73b4525700 1 -- 172.21.15.91:0/515177656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73ac0b4150 msgr2=0x7f73ac06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:02.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.208+0000 7f73b4525700 1 --2- 172.21.15.91:0/515177656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73ac0b4150 0x7f73ac06de50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009840 tx=0x7f73a8009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:02.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.208+0000 7f73b4525700 1 -- 172.21.15.91:0/515177656 shutdown_connections 2023-11-24T06:08:02.213 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.208+0000 7f73b4525700 1 --2- 172.21.15.91:0/515177656 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73ac0b5ab0 0x7f73ac06e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.213 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.208+0000 7f73b4525700 1 --2- 172.21.15.91:0/515177656 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73ac0b4150 0x7f73ac06de50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.213 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.208+0000 7f73b4525700 1 -- 172.21.15.91:0/515177656 >> 172.21.15.91:0/515177656 conn(0x7f73ac0b2740 msgr2=0x7f73ac0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:02.213 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.208+0000 7f73b4525700 1 -- 172.21.15.91:0/515177656 shutdown_connections 2023-11-24T06:08:02.213 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.208+0000 7f73b4525700 1 -- 172.21.15.91:0/515177656 wait complete. 2023-11-24T06:08:02.215 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.209+0000 7f73b4525700 1 Processor -- start 2023-11-24T06:08:02.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.209+0000 7f73b4525700 1 -- start start 2023-11-24T06:08:02.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.210+0000 7f73b4525700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73ac06b620 0x7f73ac0c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:02.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.210+0000 7f73b4525700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f73ac0b4150 0x7f73ac141640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:02.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.210+0000 7f73b4525700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73ac0b5ab0 0x7f73ac1e6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:02.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.210+0000 7f73b4525700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac141b80 con 0x7f73ac0b4150 2023-11-24T06:08:02.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.210+0000 7f73b4525700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac141db0 con 0x7f73ac06b620 2023-11-24T06:08:02.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.210+0000 7f73b4525700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac142090 con 0x7f73ac0b5ab0 2023-11-24T06:08:02.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.210+0000 7f73b1ac0700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f73ac0b4150 0x7f73ac141640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:02.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.210+0000 7f73b1ac0700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f73ac0b4150 0x7f73ac141640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57668/0 (socket says 172.21.15.91:57668) 2023-11-24T06:08:02.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.210+0000 7f73b1ac0700 1 -- 172.21.15.91:0/2086356351 learned_addr learned my addr 172.21.15.91:0/2086356351 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:02.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.210+0000 7f73b22c1700 1 --2- 172.21.15.91:0/2086356351 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73ac06b620 0x7f73ac0c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:02.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.210+0000 7f73b2ac2700 1 --2- 172.21.15.91:0/2086356351 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73ac0b5ab0 0x7f73ac1e6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:02.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.210+0000 7f73b1ac0700 1 -- 172.21.15.91:0/2086356351 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73ac0b5ab0 msgr2=0x7f73ac1e6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:02.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.210+0000 7f73b1ac0700 1 --2- 172.21.15.91:0/2086356351 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73ac0b5ab0 0x7f73ac1e6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.210+0000 7f73b1ac0700 1 -- 172.21.15.91:0/2086356351 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73ac06b620 msgr2=0x7f73ac0c63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:02.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.210+0000 7f73b1ac0700 1 --2- 172.21.15.91:0/2086356351 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73ac06b620 0x7f73ac0c63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.210+0000 7f73b1ac0700 1 -- 172.21.15.91:0/2086356351 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a4009b30 con 0x7f73ac0b4150 2023-11-24T06:08:02.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.211+0000 7f73b1ac0700 1 --2- 172.21.15.91:0/2086356351 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f73ac0b4150 0x7f73ac141640 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009fd0 tx=0x7f73a8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:02.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.211+0000 7f73a2ffd700 1 -- 172.21.15.91:0/2086356351 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a80036a0 con 0x7f73ac0b4150 2023-11-24T06:08:02.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.211+0000 7f73a2ffd700 1 -- 172.21.15.91:0/2086356351 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a801f070 con 0x7f73ac0b4150 2023-11-24T06:08:02.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.211+0000 7f73b4525700 1 -- 172.21.15.91:0/2086356351 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a80094a0 con 0x7f73ac0b4150 2023-11-24T06:08:02.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.211+0000 7f73a2ffd700 1 -- 172.21.15.91:0/2086356351 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a8003a60 con 0x7f73ac0b4150 2023-11-24T06:08:02.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.211+0000 7f73b4525700 1 -- 172.21.15.91:0/2086356351 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ac142650 con 0x7f73ac0b4150 2023-11-24T06:08:02.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.212+0000 7f73b4525700 1 -- 172.21.15.91:0/2086356351 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73ac068670 con 0x7f73ac0b4150 2023-11-24T06:08:02.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.212+0000 7f73a2ffd700 1 -- 172.21.15.91:0/2086356351 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f73a8003bc0 con 0x7f73ac0b4150 2023-11-24T06:08:02.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.212+0000 7f73a2ffd700 1 --2- 172.21.15.91:0/2086356351 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f739006aed0 0x7f739006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:02.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.213+0000 7f73a2ffd700 1 -- 172.21.15.91:0/2086356351 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73a8021070 con 0x7f73ac0b4150 2023-11-24T06:08:02.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.213+0000 7f73b22c1700 1 --2- 172.21.15.91:0/2086356351 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f739006aed0 0x7f739006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:02.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.213+0000 7f73b22c1700 1 --2- 172.21.15.91:0/2086356351 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f739006aed0 0x7f739006d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f739c007fd0 tx=0x7f739c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:02.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.217+0000 7f73a2ffd700 1 -- 172.21.15.91:0/2086356351 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73a8091080 con 0x7f73ac0b4150 2023-11-24T06:08:02.417 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.414+0000 7f73b4525700 1 -- 172.21.15.91:0/2086356351 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73ac143bb0 con 0x7f73ac0b4150 2023-11-24T06:08:02.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.416+0000 7f73a2ffd700 1 -- 172.21.15.91:0/2086356351 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16434 (secure 0 0 0) 0x7f73a8015020 con 0x7f73ac0b4150 2023-11-24T06:08:02.419 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:02.419 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":19,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:00.798347+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:02.421 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.418+0000 7f73b4525700 1 -- 172.21.15.91:0/2086356351 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f739006aed0 msgr2=0x7f739006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:02.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.418+0000 7f73b4525700 1 --2- 172.21.15.91:0/2086356351 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f739006aed0 0x7f739006d390 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f739c007fd0 tx=0x7f739c009040 comp rx=0 tx=0).stop 2023-11-24T06:08:02.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.418+0000 7f73b4525700 1 -- 172.21.15.91:0/2086356351 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f73ac0b4150 msgr2=0x7f73ac141640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:02.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.418+0000 7f73b4525700 1 --2- 172.21.15.91:0/2086356351 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f73ac0b4150 0x7f73ac141640 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f73a8009fd0 tx=0x7f73a8004750 comp rx=0 tx=0).stop 2023-11-24T06:08:02.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.418+0000 7f73b4525700 1 -- 172.21.15.91:0/2086356351 shutdown_connections 2023-11-24T06:08:02.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.418+0000 7f73b4525700 1 --2- 172.21.15.91:0/2086356351 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f739006aed0 0x7f739006d390 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.418+0000 7f73b4525700 1 --2- 172.21.15.91:0/2086356351 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73ac06b620 0x7f73ac0c63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.418+0000 7f73b4525700 1 --2- 172.21.15.91:0/2086356351 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73ac0b5ab0 0x7f73ac1e6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.418+0000 7f73b4525700 1 --2- 172.21.15.91:0/2086356351 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f73ac0b4150 0x7f73ac141640 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.418+0000 7f73b4525700 1 -- 172.21.15.91:0/2086356351 >> 172.21.15.91:0/2086356351 conn(0x7f73ac0b2740 msgr2=0x7f73ac1475e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:02.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.418+0000 7f73b4525700 1 -- 172.21.15.91:0/2086356351 shutdown_connections 2023-11-24T06:08:02.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.418+0000 7f73b4525700 1 -- 172.21.15.91:0/2086356351 wait complete. 2023-11-24T06:08:02.435 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T06:08:02.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.555+0000 7fb3bb409700 1 Processor -- start 2023-11-24T06:08:02.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.555+0000 7fb3bb409700 1 -- start start 2023-11-24T06:08:02.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.556+0000 7fb3bb409700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb3b406b620 0x7fb3b40b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:02.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.556+0000 7fb3bb409700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b41629c0 con 0x7fb3b406ee60 2023-11-24T06:08:02.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.556+0000 7fb3bb409700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b4071f50 con 0x7fb3b40b5ab0 2023-11-24T06:08:02.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.556+0000 7fb3bb409700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b4072230 con 0x7fb3b406b620 2023-11-24T06:08:02.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.556+0000 7fb3b99a6700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fb3b406ee60 0x7fb3b41622b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52612/0 (socket says 172.21.15.91:52612) 2023-11-24T06:08:02.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.556+0000 7fb3b99a6700 1 -- 172.21.15.91:0/3372285907 learned_addr learned my addr 172.21.15.91:0/3372285907 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:02.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.556+0000 7fb3b91a5700 1 --2- 172.21.15.91:0/3372285907 >> v2:172.21.15.94:3300/0 conn(0x7fb3b406b620 0x7fb3b40b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:02.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.556+0000 7fb3abfff700 1 -- 172.21.15.91:0/3372285907 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275144248 0 0) 0x7fb3b41629c0 con 0x7fb3b406ee60 2023-11-24T06:08:02.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.556+0000 7fb3abfff700 1 -- 172.21.15.91:0/3372285907 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3a00035b0 con 0x7fb3b406ee60 2023-11-24T06:08:02.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.556+0000 7fb3abfff700 1 -- 172.21.15.91:0/3372285907 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4142190857 0 0) 0x7fb3a00035b0 con 0x7fb3b406ee60 2023-11-24T06:08:02.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.556+0000 7fb3abfff700 1 -- 172.21.15.91:0/3372285907 >> v2:172.21.15.94:3300/0 conn(0x7fb3b406b620 msgr2=0x7fb3b40b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:02.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.556+0000 7fb3abfff700 1 --2- 172.21.15.91:0/3372285907 >> v2:172.21.15.94:3300/0 conn(0x7fb3b406b620 0x7fb3b40b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.556+0000 7fb3abfff700 1 -- 172.21.15.91:0/3372285907 >> v1:172.21.15.94:6789/0 conn(0x7fb3b40b5ab0 legacy=0x7fb3b4161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:02.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.556+0000 7fb3abfff700 1 -- 172.21.15.91:0/3372285907 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b4072510 con 0x7fb3b406ee60 2023-11-24T06:08:02.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3abfff700 1 -- 172.21.15.91:0/3372285907 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7fb3ac0019f0 con 0x7fb3b406ee60 2023-11-24T06:08:02.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3abfff700 1 -- 172.21.15.91:0/3372285907 >> v1:172.21.15.91:6789/0 conn(0x7fb3b406ee60 legacy=0x7fb3b41622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:02.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3abfff700 1 --2- 172.21.15.91:0/3372285907 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb3a0003a00 0x7fb3a0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:02.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3abfff700 1 --2- 172.21.15.91:0/3372285907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3a0004850 0x7fb3a0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:02.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3abfff700 1 --2- 172.21.15.91:0/3372285907 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb3a0005210 0x7fb3a000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:02.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3abfff700 1 -- 172.21.15.91:0/3372285907 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b4072230 con 0x7fb3a0003a00 2023-11-24T06:08:02.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3abfff700 1 -- 172.21.15.91:0/3372285907 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a000c0b0 con 0x7fb3a0004850 2023-11-24T06:08:02.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3abfff700 1 -- 172.21.15.91:0/3372285907 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a000c270 con 0x7fb3a0005210 2023-11-24T06:08:02.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3b91a5700 1 --2- 172.21.15.91:0/3372285907 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb3a0003a00 0x7fb3a0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:02.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3b89a4700 1 --2- 172.21.15.91:0/3372285907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3a0004850 0x7fb3a0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:02.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3b99a6700 1 --2- 172.21.15.91:0/3372285907 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb3a0005210 0x7fb3a000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:02.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3b91a5700 1 -- 172.21.15.91:0/3372285907 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb3a0005210 msgr2=0x7fb3a000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:02.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3b91a5700 1 --2- 172.21.15.91:0/3372285907 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb3a0005210 0x7fb3a000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3b91a5700 1 -- 172.21.15.91:0/3372285907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3a0004850 msgr2=0x7fb3a0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:02.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3b91a5700 1 --2- 172.21.15.91:0/3372285907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3a0004850 0x7fb3a0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3b99a6700 1 -- 172.21.15.91:0/3372285907 reap_dead start 2023-11-24T06:08:02.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3b91a5700 1 -- 172.21.15.91:0/3372285907 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a00035b0 con 0x7fb3a0003a00 2023-11-24T06:08:02.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3b89a4700 1 --2- 172.21.15.91:0/3372285907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3a0004850 0x7fb3a0004cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:02.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3b91a5700 1 --2- 172.21.15.91:0/3372285907 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb3a0003a00 0x7fb3a0003e20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4009b60 tx=0x7fb3a0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769f68e95cc591eb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:02.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3abfff700 1 -- 172.21.15.91:0/3372285907 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a400e6c0 con 0x7fb3a0003a00 2023-11-24T06:08:02.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3abfff700 1 -- 172.21.15.91:0/3372285907 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a400e820 con 0x7fb3a0003a00 2023-11-24T06:08:02.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.557+0000 7fb3abfff700 1 -- 172.21.15.91:0/3372285907 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a400eb90 con 0x7fb3a0003a00 2023-11-24T06:08:02.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.558+0000 7fb3bb409700 1 -- 172.21.15.91:0/3372285907 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb3a0003a00 msgr2=0x7fb3a0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:02.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.558+0000 7fb3bb409700 1 --2- 172.21.15.91:0/3372285907 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb3a0003a00 0x7fb3a0003e20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4009b60 tx=0x7fb3a0005880 comp rx=0 tx=0).stop 2023-11-24T06:08:02.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.558+0000 7fb3bb409700 1 -- 172.21.15.91:0/3372285907 shutdown_connections 2023-11-24T06:08:02.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.558+0000 7fb3bb409700 1 --2- 172.21.15.91:0/3372285907 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb3a0003a00 0x7fb3a0003e20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.558+0000 7fb3bb409700 1 -- 172.21.15.91:0/3372285907 >> 172.21.15.91:0/3372285907 conn(0x7fb3b40b2740 msgr2=0x7fb3b40b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:02.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.558+0000 7fb3bb409700 1 -- 172.21.15.91:0/3372285907 shutdown_connections 2023-11-24T06:08:02.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.558+0000 7fb3bb409700 1 -- 172.21.15.91:0/3372285907 wait complete. 2023-11-24T06:08:02.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.559+0000 7fb3bb409700 1 Processor -- start 2023-11-24T06:08:02.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.559+0000 7fb3bb409700 1 -- start start 2023-11-24T06:08:02.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3bb409700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb3a0003a00 0x7fb3b40b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:02.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3bb409700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3b4141540 0x7fb3b4143640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:02.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3bb409700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb3b41f3d70 0x7fb3b41f4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:02.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3bb409700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b4142690 con 0x7fb3b41f3d70 2023-11-24T06:08:02.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3bb409700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b4141a10 con 0x7fb3b4141540 2023-11-24T06:08:02.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3bb409700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b4141cf0 con 0x7fb3a0003a00 2023-11-24T06:08:02.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3b99a6700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb3b41f3d70 0x7fb3b41f4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:02.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3b99a6700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb3b41f3d70 0x7fb3b41f4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57690/0 (socket says 172.21.15.91:57690) 2023-11-24T06:08:02.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3b99a6700 1 -- 172.21.15.91:0/7017128 learned_addr learned my addr 172.21.15.91:0/7017128 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:02.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3b89a4700 1 --2- 172.21.15.91:0/7017128 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3b4141540 0x7fb3b4143640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:02.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3b91a5700 1 --2- 172.21.15.91:0/7017128 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb3a0003a00 0x7fb3b40b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:02.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3b99a6700 1 -- 172.21.15.91:0/7017128 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb3a0003a00 msgr2=0x7fb3b40b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:02.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3b99a6700 1 --2- 172.21.15.91:0/7017128 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb3a0003a00 0x7fb3b40b3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3b99a6700 1 -- 172.21.15.91:0/7017128 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3b4141540 msgr2=0x7fb3b4143640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:02.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3b99a6700 1 --2- 172.21.15.91:0/7017128 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3b4141540 0x7fb3b4143640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:02.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3b99a6700 1 -- 172.21.15.91:0/7017128 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b4141fd0 con 0x7fb3b41f3d70 2023-11-24T06:08:02.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3b99a6700 1 --2- 172.21.15.91:0/7017128 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb3b41f3d70 0x7fb3b41f4200 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4062c70 tx=0x7fb3ac00e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:02.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3a9ffb700 1 -- 172.21.15.91:0/7017128 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac00eb80 con 0x7fb3b41f3d70 2023-11-24T06:08:02.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3a9ffb700 1 -- 172.21.15.91:0/7017128 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ac00ece0 con 0x7fb3b41f3d70 2023-11-24T06:08:02.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3bb409700 1 -- 172.21.15.91:0/7017128 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b41422b0 con 0x7fb3b41f3d70 2023-11-24T06:08:02.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.560+0000 7fb3a9ffb700 1 -- 172.21.15.91:0/7017128 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac013660 con 0x7fb3b41f3d70 2023-11-24T06:08:02.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.561+0000 7fb3bb409700 1 -- 172.21.15.91:0/7017128 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b40b41c0 con 0x7fb3b41f3d70 2023-11-24T06:08:02.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.561+0000 7fb3bb409700 1 -- 172.21.15.91:0/7017128 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b4067a40 con 0x7fb3b41f3d70 2023-11-24T06:08:02.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.562+0000 7fb3a9ffb700 1 -- 172.21.15.91:0/7017128 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb3ac0137c0 con 0x7fb3b41f3d70 2023-11-24T06:08:02.572 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.562+0000 7fb3a9ffb700 1 --2- 172.21.15.91:0/7017128 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb39806aed0 0x7fb39806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:02.572 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.562+0000 7fb3a9ffb700 1 -- 172.21.15.91:0/7017128 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3ac0897d0 con 0x7fb3b41f3d70 2023-11-24T06:08:02.572 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.562+0000 7fb3b91a5700 1 --2- 172.21.15.91:0/7017128 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb39806aed0 0x7fb39806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:02.572 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.563+0000 7fb3b91a5700 1 --2- 172.21.15.91:0/7017128 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb39806aed0 0x7fb39806d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40090b0 tx=0x7fb3a401e040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:02.572 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.567+0000 7fb3a9ffb700 1 -- 172.21.15.91:0/7017128 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3b4067a40 con 0x7fb3b41f3d70 2023-11-24T06:08:02.804 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:02.801+0000 7fb3bb409700 1 -- 172.21.15.91:0/7017128 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb3b41427d0 con 0x7fb3b41f3d70 2023-11-24T06:08:03.738 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.734+0000 7fb3a9ffb700 1 -- 172.21.15.91:0/7017128 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb3b41427d0 con 0x7fb3b41f3d70 2023-11-24T06:08:03.741 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.737+0000 7fb3bb409700 1 -- 172.21.15.91:0/7017128 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb39806aed0 msgr2=0x7fb39806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:03.741 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.737+0000 7fb3bb409700 1 --2- 172.21.15.91:0/7017128 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb39806aed0 0x7fb39806d390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40090b0 tx=0x7fb3a401e040 comp rx=0 tx=0).stop 2023-11-24T06:08:03.741 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.737+0000 7fb3bb409700 1 -- 172.21.15.91:0/7017128 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb3b41f3d70 msgr2=0x7fb3b41f4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:03.741 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.737+0000 7fb3bb409700 1 --2- 172.21.15.91:0/7017128 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb3b41f3d70 0x7fb3b41f4200 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4062c70 tx=0x7fb3ac00e440 comp rx=0 tx=0).stop 2023-11-24T06:08:03.741 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.737+0000 7fb3bb409700 1 -- 172.21.15.91:0/7017128 shutdown_connections 2023-11-24T06:08:03.742 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.737+0000 7fb3bb409700 1 --2- 172.21.15.91:0/7017128 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb39806aed0 0x7fb39806d390 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:03.742 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.737+0000 7fb3bb409700 1 --2- 172.21.15.91:0/7017128 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb3b41f3d70 0x7fb3b41f4200 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:03.742 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.737+0000 7fb3bb409700 1 --2- 172.21.15.91:0/7017128 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb3a0003a00 0x7fb3b40b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:03.742 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.737+0000 7fb3bb409700 1 --2- 172.21.15.91:0/7017128 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb3b4141540 0x7fb3b4143640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:03.742 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.737+0000 7fb3bb409700 1 -- 172.21.15.91:0/7017128 >> 172.21.15.91:0/7017128 conn(0x7fb3b40b2740 msgr2=0x7fb3b40b1350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:03.742 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.737+0000 7fb3bb409700 1 -- 172.21.15.91:0/7017128 shutdown_connections 2023-11-24T06:08:03.742 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.737+0000 7fb3bb409700 1 -- 172.21.15.91:0/7017128 wait complete. 2023-11-24T06:08:03.754 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T06:08:03.755 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T06:08:03.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.879+0000 7f65a304b700 1 Processor -- start 2023-11-24T06:08:03.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.879+0000 7f65a304b700 1 -- start start 2023-11-24T06:08:03.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.879+0000 7f65a304b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f659c06b620 0x7f659c0b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:03.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f65a304b700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659c1629c0 con 0x7f659c06ee60 2023-11-24T06:08:03.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f65a304b700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659c071f50 con 0x7f659c0b5ab0 2023-11-24T06:08:03.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f65a304b700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f659c072230 con 0x7f659c06b620 2023-11-24T06:08:03.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f65a15e8700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f659c06ee60 0x7f659c1622b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52632/0 (socket says 172.21.15.91:52632) 2023-11-24T06:08:03.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f65a15e8700 1 -- 172.21.15.91:0/2089198594 learned_addr learned my addr 172.21.15.91:0/2089198594 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:03.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f65a0de7700 1 --2- 172.21.15.91:0/2089198594 >> v2:172.21.15.94:3300/0 conn(0x7f659c06b620 0x7f659c0b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:03.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f659b7fe700 1 -- 172.21.15.91:0/2089198594 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401558760 0 0) 0x7f659c1629c0 con 0x7f659c06ee60 2023-11-24T06:08:03.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f659b7fe700 1 -- 172.21.15.91:0/2089198594 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f658c0035b0 con 0x7f659c06ee60 2023-11-24T06:08:03.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f659b7fe700 1 -- 172.21.15.91:0/2089198594 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1581098321 0 0) 0x7f658c0035b0 con 0x7f659c06ee60 2023-11-24T06:08:03.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f659b7fe700 1 -- 172.21.15.91:0/2089198594 >> v2:172.21.15.94:3300/0 conn(0x7f659c06b620 msgr2=0x7f659c0b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:03.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f659b7fe700 1 --2- 172.21.15.91:0/2089198594 >> v2:172.21.15.94:3300/0 conn(0x7f659c06b620 0x7f659c0b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:03.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f659b7fe700 1 -- 172.21.15.91:0/2089198594 >> v1:172.21.15.94:6789/0 conn(0x7f659c0b5ab0 legacy=0x7f659c161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:03.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f659b7fe700 1 -- 172.21.15.91:0/2089198594 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c072510 con 0x7f659c06ee60 2023-11-24T06:08:03.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f659b7fe700 1 -- 172.21.15.91:0/2089198594 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f65940019f0 con 0x7f659c06ee60 2023-11-24T06:08:03.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f659b7fe700 1 -- 172.21.15.91:0/2089198594 >> v1:172.21.15.91:6789/0 conn(0x7f659c06ee60 legacy=0x7f659c1622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:03.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f659b7fe700 1 --2- 172.21.15.91:0/2089198594 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658c003a00 0x7f658c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:03.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f659b7fe700 1 --2- 172.21.15.91:0/2089198594 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f658c004850 0x7f658c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:03.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f659b7fe700 1 --2- 172.21.15.91:0/2089198594 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f658c005210 0x7f658c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:03.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f659b7fe700 1 -- 172.21.15.91:0/2089198594 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c072230 con 0x7f658c004850 2023-11-24T06:08:03.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f659b7fe700 1 -- 172.21.15.91:0/2089198594 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c00c0b0 con 0x7f658c003a00 2023-11-24T06:08:03.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f659b7fe700 1 -- 172.21.15.91:0/2089198594 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658c00c270 con 0x7f658c005210 2023-11-24T06:08:03.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f659bfff700 1 --2- 172.21.15.91:0/2089198594 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f658c004850 0x7f658c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:03.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f65a0de7700 1 --2- 172.21.15.91:0/2089198594 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658c003a00 0x7f658c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:03.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.880+0000 7f65a15e8700 1 --2- 172.21.15.91:0/2089198594 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f658c005210 0x7f658c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:03.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f659bfff700 1 -- 172.21.15.91:0/2089198594 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f658c005210 msgr2=0x7f658c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:03.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f659bfff700 1 --2- 172.21.15.91:0/2089198594 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f658c005210 0x7f658c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:03.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f659bfff700 1 -- 172.21.15.91:0/2089198594 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658c003a00 msgr2=0x7f658c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:03.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f659bfff700 1 --2- 172.21.15.91:0/2089198594 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f658c003a00 0x7f658c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:03.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f659bfff700 1 -- 172.21.15.91:0/2089198594 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c0035b0 con 0x7f658c004850 2023-11-24T06:08:03.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f65a15e8700 1 --2- 172.21.15.91:0/2089198594 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f658c005210 0x7f658c00ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:03.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f65a15e8700 1 -- 172.21.15.91:0/2089198594 reap_dead start 2023-11-24T06:08:03.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f659bfff700 1 --2- 172.21.15.91:0/2089198594 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f658c004850 0x7f658c004cd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f659c0b5f10 tx=0x7f658c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f391cd6c045c36c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:03.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f659b7fe700 1 -- 172.21.15.91:0/2089198594 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65880052d0 con 0x7f658c004850 2023-11-24T06:08:03.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f659b7fe700 1 -- 172.21.15.91:0/2089198594 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6588005430 con 0x7f658c004850 2023-11-24T06:08:03.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f659b7fe700 1 -- 172.21.15.91:0/2089198594 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65880038c0 con 0x7f658c004850 2023-11-24T06:08:03.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f65a304b700 1 -- 172.21.15.91:0/2089198594 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f658c004850 msgr2=0x7f658c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:03.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f65a304b700 1 --2- 172.21.15.91:0/2089198594 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f658c004850 0x7f658c004cd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f659c0b5f10 tx=0x7f658c005880 comp rx=0 tx=0).stop 2023-11-24T06:08:03.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f65a304b700 1 -- 172.21.15.91:0/2089198594 shutdown_connections 2023-11-24T06:08:03.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f65a304b700 1 --2- 172.21.15.91:0/2089198594 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f658c004850 0x7f658c004cd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:03.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f65a304b700 1 -- 172.21.15.91:0/2089198594 >> 172.21.15.91:0/2089198594 conn(0x7f659c0b2740 msgr2=0x7f659c0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:03.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f65a304b700 1 -- 172.21.15.91:0/2089198594 shutdown_connections 2023-11-24T06:08:03.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.881+0000 7f65a304b700 1 -- 172.21.15.91:0/2089198594 wait complete. 2023-11-24T06:08:03.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.883+0000 7f65a304b700 1 Processor -- start 2023-11-24T06:08:03.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.883+0000 7f65a304b700 1 -- start start 2023-11-24T06:08:03.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.883+0000 7f65a304b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f658c004850 0x7f659c143390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:03.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.883+0000 7f65a304b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659c161460 0x7f659c1618e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:03.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.883+0000 7f65a304b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f659c1f3dd0 0x7f659c1f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:03.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65a304b700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c0c5e00 con 0x7f659c1f3dd0 2023-11-24T06:08:03.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65a304b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c161e20 con 0x7f659c161460 2023-11-24T06:08:03.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65a304b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659c1438d0 con 0x7f658c004850 2023-11-24T06:08:03.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65a15e8700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f659c1f3dd0 0x7f659c1f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:03.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65a15e8700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f659c1f3dd0 0x7f659c1f4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57710/0 (socket says 172.21.15.91:57710) 2023-11-24T06:08:03.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65a15e8700 1 -- 172.21.15.91:0/3182494423 learned_addr learned my addr 172.21.15.91:0/3182494423 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:03.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65a0de7700 1 --2- 172.21.15.91:0/3182494423 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f658c004850 0x7f659c143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:03.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f659bfff700 1 --2- 172.21.15.91:0/3182494423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659c161460 0x7f659c1618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:03.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65a15e8700 1 -- 172.21.15.91:0/3182494423 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f658c004850 msgr2=0x7f659c143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:03.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65a15e8700 1 --2- 172.21.15.91:0/3182494423 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f658c004850 0x7f659c143390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:03.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65a15e8700 1 -- 172.21.15.91:0/3182494423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659c161460 msgr2=0x7f659c1618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:03.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65a15e8700 1 --2- 172.21.15.91:0/3182494423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659c161460 0x7f659c1618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:03.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65a15e8700 1 -- 172.21.15.91:0/3182494423 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c141640 con 0x7f659c1f3dd0 2023-11-24T06:08:03.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f659bfff700 1 --2- 172.21.15.91:0/3182494423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659c161460 0x7f659c1618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:03.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65a15e8700 1 --2- 172.21.15.91:0/3182494423 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f659c1f3dd0 0x7f659c1f4250 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f659c073020 tx=0x7f6594007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:03.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65997fa700 1 -- 172.21.15.91:0/3182494423 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659400eba0 con 0x7f659c1f3dd0 2023-11-24T06:08:03.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65997fa700 1 -- 172.21.15.91:0/3182494423 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659400ed00 con 0x7f659c1f3dd0 2023-11-24T06:08:03.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65a304b700 1 -- 172.21.15.91:0/3182494423 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659c141920 con 0x7f659c1f3dd0 2023-11-24T06:08:03.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65997fa700 1 -- 172.21.15.91:0/3182494423 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594013660 con 0x7f659c1f3dd0 2023-11-24T06:08:03.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.884+0000 7f65a304b700 1 -- 172.21.15.91:0/3182494423 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659c141ee0 con 0x7f659c1f3dd0 2023-11-24T06:08:03.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.885+0000 7f65a304b700 1 -- 172.21.15.91:0/3182494423 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659c067a40 con 0x7f659c1f3dd0 2023-11-24T06:08:03.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.886+0000 7f65997fa700 1 -- 172.21.15.91:0/3182494423 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f65940137c0 con 0x7f659c1f3dd0 2023-11-24T06:08:03.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.886+0000 7f65997fa700 1 --2- 172.21.15.91:0/3182494423 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f658006aed0 0x7f658006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:03.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.886+0000 7f65a0de7700 1 --2- 172.21.15.91:0/3182494423 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f658006aed0 0x7f658006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:03.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.886+0000 7f65997fa700 1 -- 172.21.15.91:0/3182494423 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6594089ac0 con 0x7f659c1f3dd0 2023-11-24T06:08:03.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.886+0000 7f65a0de7700 1 --2- 172.21.15.91:0/3182494423 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f658006aed0 0x7f658006d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6590009ce0 tx=0x7f6590009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:03.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:03.890+0000 7f65997fa700 1 -- 172.21.15.91:0/3182494423 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f659c067a40 con 0x7f659c1f3dd0 2023-11-24T06:08:03.994 INFO:tasks.ceph.mds.a.smithi091.stderr:2023-11-24T06:08:03.990+0000 7f8430f69700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T06:08:04.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.130+0000 7f65a304b700 1 -- 172.21.15.91:0/3182494423 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f659c067a40 con 0x7f659c1f3dd0 2023-11-24T06:08:04.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.735+0000 7f65997fa700 1 -- 172.21.15.91:0/3182494423 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f659c067a40 con 0x7f659c1f3dd0 2023-11-24T06:08:04.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.738+0000 7f65a304b700 1 -- 172.21.15.91:0/3182494423 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f658006aed0 msgr2=0x7f658006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:04.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.738+0000 7f65a304b700 1 --2- 172.21.15.91:0/3182494423 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f658006aed0 0x7f658006d390 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6590009ce0 tx=0x7f6590009210 comp rx=0 tx=0).stop 2023-11-24T06:08:04.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.738+0000 7f65a304b700 1 -- 172.21.15.91:0/3182494423 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f659c1f3dd0 msgr2=0x7f659c1f4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:04.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.738+0000 7f65a304b700 1 --2- 172.21.15.91:0/3182494423 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f659c1f3dd0 0x7f659c1f4250 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f659c073020 tx=0x7f6594007c90 comp rx=0 tx=0).stop 2023-11-24T06:08:04.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.739+0000 7f65a304b700 1 -- 172.21.15.91:0/3182494423 shutdown_connections 2023-11-24T06:08:04.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.739+0000 7f65a304b700 1 --2- 172.21.15.91:0/3182494423 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f658006aed0 0x7f658006d390 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:04.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.739+0000 7f65a304b700 1 --2- 172.21.15.91:0/3182494423 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f659c1f3dd0 0x7f659c1f4250 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:04.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.739+0000 7f65a304b700 1 --2- 172.21.15.91:0/3182494423 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f658c004850 0x7f659c143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:04.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.739+0000 7f65a304b700 1 --2- 172.21.15.91:0/3182494423 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f659c161460 0x7f659c1618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:04.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.739+0000 7f65a304b700 1 -- 172.21.15.91:0/3182494423 >> 172.21.15.91:0/3182494423 conn(0x7f659c0b2740 msgr2=0x7f659c0b09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:04.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.739+0000 7f65a304b700 1 -- 172.21.15.91:0/3182494423 shutdown_connections 2023-11-24T06:08:04.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.739+0000 7f65a304b700 1 -- 172.21.15.91:0/3182494423 wait complete. 2023-11-24T06:08:04.757 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T06:08:04.757 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:08:04.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.879+0000 7f55b8c9d700 1 Processor -- start 2023-11-24T06:08:04.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.879+0000 7f55b8c9d700 1 -- start start 2023-11-24T06:08:04.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.880+0000 7f55b8c9d700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f55b406afd0 0x7f55b40b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:04.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.880+0000 7f55b8c9d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f55b406edc0 0x7f55b406b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:04.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.880+0000 7f55b8c9d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55b406dba0 0x7f55b406e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:04.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.880+0000 7f55b8c9d700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b406f830 con 0x7f55b406edc0 2023-11-24T06:08:04.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.880+0000 7f55b8c9d700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b40ad860 con 0x7f55b406afd0 2023-11-24T06:08:04.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.880+0000 7f55b8c9d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40d0a20 con 0x7f55b406dba0 2023-11-24T06:08:04.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.880+0000 7f55b1d9b700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f55b406edc0 0x7f55b406b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:04.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.880+0000 7f55b1d9b700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f55b406edc0 0x7f55b406b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57714/0 (socket says 172.21.15.91:57714) 2023-11-24T06:08:04.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.880+0000 7f55b1d9b700 1 -- 172.21.15.91:0/3349987307 learned_addr learned my addr 172.21.15.91:0/3349987307 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:04.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.881+0000 7f55b2d9d700 1 --2- 172.21.15.91:0/3349987307 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55b406dba0 0x7f55b406e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:04.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.881+0000 7f55b259c700 1 --2- 172.21.15.91:0/3349987307 >> v2:172.21.15.94:3300/0 conn(0x7f55b406afd0 0x7f55b40b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:04.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.881+0000 7f55b1d9b700 1 -- 172.21.15.91:0/3349987307 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55b406dba0 msgr2=0x7f55b406e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:04.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.881+0000 7f55b1d9b700 1 --2- 172.21.15.91:0/3349987307 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55b406dba0 0x7f55b406e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:04.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.881+0000 7f55b1d9b700 1 -- 172.21.15.91:0/3349987307 >> v2:172.21.15.94:3300/0 conn(0x7f55b406afd0 msgr2=0x7f55b40b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:04.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.881+0000 7f55b1d9b700 1 --2- 172.21.15.91:0/3349987307 >> v2:172.21.15.94:3300/0 conn(0x7f55b406afd0 0x7f55b40b5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:04.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.881+0000 7f55b1d9b700 1 -- 172.21.15.91:0/3349987307 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b4162a70 con 0x7f55b406edc0 2023-11-24T06:08:04.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.881+0000 7f55b1d9b700 1 --2- 172.21.15.91:0/3349987307 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55b406edc0 0x7f55b406b3b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f55b4073820 tx=0x7f55a4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deb3164a042fc450 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:04.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.881+0000 7f55b159a700 1 -- 172.21.15.91:0/3349987307 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a400b870 con 0x7f55b406edc0 2023-11-24T06:08:04.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.881+0000 7f55b159a700 1 -- 172.21.15.91:0/3349987307 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a400b9d0 con 0x7f55b406edc0 2023-11-24T06:08:04.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.881+0000 7f55b159a700 1 -- 172.21.15.91:0/3349987307 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a400bd60 con 0x7f55b406edc0 2023-11-24T06:08:04.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.881+0000 7f55b8c9d700 1 -- 172.21.15.91:0/3349987307 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55b406edc0 msgr2=0x7f55b406b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:04.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.881+0000 7f55b8c9d700 1 --2- 172.21.15.91:0/3349987307 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55b406edc0 0x7f55b406b3b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f55b4073820 tx=0x7f55a4009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:04.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.882+0000 7f55b8c9d700 1 -- 172.21.15.91:0/3349987307 shutdown_connections 2023-11-24T06:08:04.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.882+0000 7f55b8c9d700 1 --2- 172.21.15.91:0/3349987307 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55b406dba0 0x7f55b406e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:04.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.882+0000 7f55b8c9d700 1 --2- 172.21.15.91:0/3349987307 >> v2:172.21.15.94:3300/0 conn(0x7f55b406afd0 0x7f55b40b5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:04.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.882+0000 7f55b8c9d700 1 --2- 172.21.15.91:0/3349987307 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55b406edc0 0x7f55b406b3b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:04.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.882+0000 7f55b8c9d700 1 -- 172.21.15.91:0/3349987307 >> 172.21.15.91:0/3349987307 conn(0x7f55b40b2840 msgr2=0x7f55b40b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:04.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.882+0000 7f55b8c9d700 1 -- 172.21.15.91:0/3349987307 shutdown_connections 2023-11-24T06:08:04.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.882+0000 7f55b8c9d700 1 -- 172.21.15.91:0/3349987307 wait complete. 2023-11-24T06:08:04.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.883+0000 7f55b8c9d700 1 Processor -- start 2023-11-24T06:08:04.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.883+0000 7f55b8c9d700 1 -- start start 2023-11-24T06:08:04.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b8c9d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55b406afd0 0x7f55b4143690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:04.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b8c9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55b406dba0 0x7f55b415ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:04.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b8c9d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55b406edc0 0x7f55b40aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:04.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b8c9d700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b4143bd0 con 0x7f55b406dba0 2023-11-24T06:08:04.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b8c9d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b4158340 con 0x7f55b406edc0 2023-11-24T06:08:04.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b8c9d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b415f5a0 con 0x7f55b406afd0 2023-11-24T06:08:04.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b1d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55b406dba0 0x7f55b415ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:04.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b1d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55b406dba0 0x7f55b415ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57722/0 (socket says 172.21.15.91:57722) 2023-11-24T06:08:04.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b1d9b700 1 -- 172.21.15.91:0/832671378 learned_addr learned my addr 172.21.15.91:0/832671378 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:04.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b259c700 1 --2- 172.21.15.91:0/832671378 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55b406afd0 0x7f55b4143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:04.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b2d9d700 1 --2- 172.21.15.91:0/832671378 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55b406edc0 0x7f55b40aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:04.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b1d9b700 1 -- 172.21.15.91:0/832671378 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55b406afd0 msgr2=0x7f55b4143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:04.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b1d9b700 1 --2- 172.21.15.91:0/832671378 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55b406afd0 0x7f55b4143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:04.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b1d9b700 1 -- 172.21.15.91:0/832671378 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55b406edc0 msgr2=0x7f55b40aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:04.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b1d9b700 1 --2- 172.21.15.91:0/832671378 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55b406edc0 0x7f55b40aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:04.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b1d9b700 1 -- 172.21.15.91:0/832671378 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a4009550 con 0x7f55b406dba0 2023-11-24T06:08:04.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b2d9d700 1 --2- 172.21.15.91:0/832671378 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55b406edc0 0x7f55b40aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:04.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b1d9b700 1 --2- 172.21.15.91:0/832671378 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55b406dba0 0x7f55b415ec50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f55a4000c00 tx=0x7f55a4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:04.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55a2ffd700 1 -- 172.21.15.91:0/832671378 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a4004460 con 0x7f55b406dba0 2023-11-24T06:08:04.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55a2ffd700 1 -- 172.21.15.91:0/832671378 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a40045c0 con 0x7f55b406dba0 2023-11-24T06:08:04.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55b8c9d700 1 -- 172.21.15.91:0/832671378 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b4158590 con 0x7f55b406dba0 2023-11-24T06:08:04.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.884+0000 7f55a2ffd700 1 -- 172.21.15.91:0/832671378 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a40038a0 con 0x7f55b406dba0 2023-11-24T06:08:04.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.885+0000 7f55b8c9d700 1 -- 172.21.15.91:0/832671378 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b4158b50 con 0x7f55b406dba0 2023-11-24T06:08:04.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.885+0000 7f55b8c9d700 1 -- 172.21.15.91:0/832671378 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b4158ec0 con 0x7f55b406dba0 2023-11-24T06:08:04.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.886+0000 7f55a2ffd700 1 -- 172.21.15.91:0/832671378 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f55a4021070 con 0x7f55b406dba0 2023-11-24T06:08:04.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.886+0000 7f55a2ffd700 1 --2- 172.21.15.91:0/832671378 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f559406aed0 0x7f559406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:04.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.886+0000 7f55a2ffd700 1 -- 172.21.15.91:0/832671378 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55a408aba0 con 0x7f55b406dba0 2023-11-24T06:08:04.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.886+0000 7f55b259c700 1 --2- 172.21.15.91:0/832671378 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f559406aed0 0x7f559406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:04.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.887+0000 7f55b259c700 1 --2- 172.21.15.91:0/832671378 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f559406aed0 0x7f559406d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f55a8009950 tx=0x7f55a8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:04.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:04.891+0000 7f55a2ffd700 1 -- 172.21.15.91:0/832671378 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f55a4055e00 con 0x7f55b406dba0 2023-11-24T06:08:05.128 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.124+0000 7f55b8c9d700 1 -- 172.21.15.91:0/832671378 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55b4158ce0 con 0x7f55b406dba0 2023-11-24T06:08:05.128 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.124+0000 7f55a2ffd700 1 -- 172.21.15.91:0/832671378 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4907 (secure 0 0 0) 0x7f55a4015090 con 0x7f55b406dba0 2023-11-24T06:08:05.128 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:05.129 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1329219967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1329219967},{"type":"v1","addr":"172.21.15.94:6834","nonce":1329219967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3439524864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3439524864},{"type":"v1","addr":"172.21.15.94:6835","nonce":3439524864}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6836/3390368219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6834","nonce":3390368219},{"type":"v1","addr":"172.21.15.91:6836","nonce":3390368219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:08:00.798681+0000","modified":"2023-11-24T06:08:04.132764+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.91:6837/1644079645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":1644079645},{"type":"v1","addr":"172.21.15.91:6837","nonce":1644079645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T06:08:05.131 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.127+0000 7f55b8c9d700 1 -- 172.21.15.91:0/832671378 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f559406aed0 msgr2=0x7f559406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.131 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.127+0000 7f55b8c9d700 1 --2- 172.21.15.91:0/832671378 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f559406aed0 0x7f559406d390 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f55a8009950 tx=0x7f55a8008040 comp rx=0 tx=0).stop 2023-11-24T06:08:05.131 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.127+0000 7f55b8c9d700 1 -- 172.21.15.91:0/832671378 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55b406dba0 msgr2=0x7f55b415ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.131 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.127+0000 7f55b8c9d700 1 --2- 172.21.15.91:0/832671378 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55b406dba0 0x7f55b415ec50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f55a4000c00 tx=0x7f55a4004750 comp rx=0 tx=0).stop 2023-11-24T06:08:05.131 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.127+0000 7f55b8c9d700 1 -- 172.21.15.91:0/832671378 shutdown_connections 2023-11-24T06:08:05.131 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.127+0000 7f55b8c9d700 1 --2- 172.21.15.91:0/832671378 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f559406aed0 0x7f559406d390 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.132 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.127+0000 7f55b8c9d700 1 --2- 172.21.15.91:0/832671378 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55b406edc0 0x7f55b40aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.132 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.127+0000 7f55b8c9d700 1 --2- 172.21.15.91:0/832671378 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55b406afd0 0x7f55b4143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.132 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.127+0000 7f55b8c9d700 1 --2- 172.21.15.91:0/832671378 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55b406dba0 0x7f55b415ec50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.132 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.127+0000 7f55b8c9d700 1 -- 172.21.15.91:0/832671378 >> 172.21.15.91:0/832671378 conn(0x7f55b40b2840 msgr2=0x7f55b40b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:05.132 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.127+0000 7f55b8c9d700 1 -- 172.21.15.91:0/832671378 shutdown_connections 2023-11-24T06:08:05.132 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.127+0000 7f55b8c9d700 1 -- 172.21.15.91:0/832671378 wait complete. 2023-11-24T06:08:05.132 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 8 2023-11-24T06:08:05.143 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:05.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.259+0000 7ff910844700 1 Processor -- start 2023-11-24T06:08:05.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.259+0000 7ff910844700 1 -- start start 2023-11-24T06:08:05.267 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff910844700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff90c0b4150 0x7ff90c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.267 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff910844700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90c071f50 con 0x7ff90c0b5ab0 2023-11-24T06:08:05.267 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff910844700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90c072230 con 0x7ff90c06b620 2023-11-24T06:08:05.267 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff910844700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff90c072510 con 0x7ff90c0b4150 2023-11-24T06:08:05.267 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90ad9d700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7ff90c0b5ab0 0x7ff90c06e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52666/0 (socket says 172.21.15.91:52666) 2023-11-24T06:08:05.267 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90ad9d700 1 -- 172.21.15.91:0/3873344686 learned_addr learned my addr 172.21.15.91:0/3873344686 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:05.267 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff909d9b700 1 --2- 172.21.15.91:0/3873344686 >> v2:172.21.15.94:3300/0 conn(0x7ff90c0b4150 0x7ff90c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.267 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90959a700 1 -- 172.21.15.91:0/3873344686 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439780548 0 0) 0x7ff90c071f50 con 0x7ff90c0b5ab0 2023-11-24T06:08:05.267 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90959a700 1 -- 172.21.15.91:0/3873344686 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f80035b0 con 0x7ff90c0b5ab0 2023-11-24T06:08:05.268 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90959a700 1 -- 172.21.15.91:0/3873344686 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2352911599 0 0) 0x7ff8f80035b0 con 0x7ff90c0b5ab0 2023-11-24T06:08:05.268 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90959a700 1 -- 172.21.15.91:0/3873344686 >> v2:172.21.15.94:3300/0 conn(0x7ff90c0b4150 msgr2=0x7ff90c06de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:05.268 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90959a700 1 --2- 172.21.15.91:0/3873344686 >> v2:172.21.15.94:3300/0 conn(0x7ff90c0b4150 0x7ff90c06de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.268 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90959a700 1 -- 172.21.15.91:0/3873344686 >> v1:172.21.15.94:6789/0 conn(0x7ff90c06b620 legacy=0x7ff90c0b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.268 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90959a700 1 -- 172.21.15.91:0/3873344686 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90c0727f0 con 0x7ff90c0b5ab0 2023-11-24T06:08:05.268 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90959a700 1 -- 172.21.15.91:0/3873344686 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7ff9040019f0 con 0x7ff90c0b5ab0 2023-11-24T06:08:05.268 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90959a700 1 -- 172.21.15.91:0/3873344686 >> v1:172.21.15.91:6789/0 conn(0x7ff90c0b5ab0 legacy=0x7ff90c06e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.268 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90959a700 1 --2- 172.21.15.91:0/3873344686 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8f8003a00 0x7ff8f8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.268 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90959a700 1 --2- 172.21.15.91:0/3873344686 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8f8004850 0x7ff8f8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.268 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90959a700 1 --2- 172.21.15.91:0/3873344686 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8f8005210 0x7ff8f800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.269 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90959a700 1 -- 172.21.15.91:0/3873344686 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c072510 con 0x7ff8f8005210 2023-11-24T06:08:05.269 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90959a700 1 -- 172.21.15.91:0/3873344686 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f800c0b0 con 0x7ff8f8003a00 2023-11-24T06:08:05.269 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90959a700 1 -- 172.21.15.91:0/3873344686 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f800c270 con 0x7ff8f8004850 2023-11-24T06:08:05.269 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90ad9d700 1 --2- 172.21.15.91:0/3873344686 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8f8005210 0x7ff8f800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.269 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff90a59c700 1 --2- 172.21.15.91:0/3873344686 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8f8003a00 0x7ff8f8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.269 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.260+0000 7ff909d9b700 1 --2- 172.21.15.91:0/3873344686 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8f8004850 0x7ff8f8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.269 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.261+0000 7ff90ad9d700 1 -- 172.21.15.91:0/3873344686 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8f8004850 msgr2=0x7ff8f8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:05.269 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.261+0000 7ff90ad9d700 1 --2- 172.21.15.91:0/3873344686 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8f8004850 0x7ff8f8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.269 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.261+0000 7ff90ad9d700 1 -- 172.21.15.91:0/3873344686 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8f8003a00 msgr2=0x7ff8f8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:05.269 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.261+0000 7ff90ad9d700 1 --2- 172.21.15.91:0/3873344686 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8f8003a00 0x7ff8f8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.269 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.261+0000 7ff90ad9d700 1 -- 172.21.15.91:0/3873344686 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f80035b0 con 0x7ff8f8005210 2023-11-24T06:08:05.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.261+0000 7ff90ad9d700 1 -- 172.21.15.91:0/3873344686 reap_dead start 2023-11-24T06:08:05.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.261+0000 7ff90ad9d700 1 --2- 172.21.15.91:0/3873344686 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8f8005210 0x7ff8f800ba10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff90400a830 tx=0x7ff8f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38ac690e9bd64dd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:05.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.261+0000 7ff90959a700 1 -- 172.21.15.91:0/3873344686 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90400a2f0 con 0x7ff8f8005210 2023-11-24T06:08:05.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.261+0000 7ff90959a700 1 -- 172.21.15.91:0/3873344686 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff904012620 con 0x7ff8f8005210 2023-11-24T06:08:05.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.261+0000 7ff90959a700 1 -- 172.21.15.91:0/3873344686 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9040129d0 con 0x7ff8f8005210 2023-11-24T06:08:05.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.261+0000 7ff910844700 1 -- 172.21.15.91:0/3873344686 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8f8005210 msgr2=0x7ff8f800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.261+0000 7ff910844700 1 --2- 172.21.15.91:0/3873344686 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8f8005210 0x7ff8f800ba10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff90400a830 tx=0x7ff8f8005880 comp rx=0 tx=0).stop 2023-11-24T06:08:05.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.262+0000 7ff910844700 1 -- 172.21.15.91:0/3873344686 shutdown_connections 2023-11-24T06:08:05.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.262+0000 7ff910844700 1 --2- 172.21.15.91:0/3873344686 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8f8005210 0x7ff8f800ba10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.262+0000 7ff910844700 1 -- 172.21.15.91:0/3873344686 >> 172.21.15.91:0/3873344686 conn(0x7ff90c0b2740 msgr2=0x7ff90c0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:05.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.262+0000 7ff910844700 1 -- 172.21.15.91:0/3873344686 shutdown_connections 2023-11-24T06:08:05.271 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.262+0000 7ff910844700 1 -- 172.21.15.91:0/3873344686 wait complete. 2023-11-24T06:08:05.271 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.263+0000 7ff910844700 1 Processor -- start 2023-11-24T06:08:05.271 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.263+0000 7ff910844700 1 -- start start 2023-11-24T06:08:05.273 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff910844700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8f8005210 0x7ff90c0c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.273 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff910844700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff90c1e80f0 0x7ff90c0c75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.273 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff910844700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff90c1e6a90 0x7ff90c144190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.273 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff910844700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c1e7c60 con 0x7ff90c1e6a90 2023-11-24T06:08:05.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff910844700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c0c6b30 con 0x7ff90c1e80f0 2023-11-24T06:08:05.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff910844700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c1e7050 con 0x7ff8f8005210 2023-11-24T06:08:05.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff90ad9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff90c1e6a90 0x7ff90c144190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff90ad9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff90c1e6a90 0x7ff90c144190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57744/0 (socket says 172.21.15.91:57744) 2023-11-24T06:08:05.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff90ad9d700 1 -- 172.21.15.91:0/746818906 learned_addr learned my addr 172.21.15.91:0/746818906 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:05.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff90a59c700 1 --2- 172.21.15.91:0/746818906 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8f8005210 0x7ff90c0c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff909d9b700 1 --2- 172.21.15.91:0/746818906 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff90c1e80f0 0x7ff90c0c75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff90ad9d700 1 -- 172.21.15.91:0/746818906 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8f8005210 msgr2=0x7ff90c0c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff90ad9d700 1 --2- 172.21.15.91:0/746818906 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8f8005210 0x7ff90c0c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff90ad9d700 1 -- 172.21.15.91:0/746818906 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff90c1e80f0 msgr2=0x7ff90c0c75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff90ad9d700 1 --2- 172.21.15.91:0/746818906 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff90c1e80f0 0x7ff90c0c75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff90ad9d700 1 -- 172.21.15.91:0/746818906 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90c1e7330 con 0x7ff90c1e6a90 2023-11-24T06:08:05.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff90ad9d700 1 --2- 172.21.15.91:0/746818906 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff90c1e6a90 0x7ff90c144190 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff90c0706e0 tx=0x7ff904001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:05.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff902ffd700 1 -- 172.21.15.91:0/746818906 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff904024070 con 0x7ff90c1e6a90 2023-11-24T06:08:05.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff902ffd700 1 -- 172.21.15.91:0/746818906 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff904001250 con 0x7ff90c1e6a90 2023-11-24T06:08:05.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff910844700 1 -- 172.21.15.91:0/746818906 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff90c1e7610 con 0x7ff90c1e6a90 2023-11-24T06:08:05.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff902ffd700 1 -- 172.21.15.91:0/746818906 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9040017b0 con 0x7ff90c1e6a90 2023-11-24T06:08:05.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.264+0000 7ff910844700 1 -- 172.21.15.91:0/746818906 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff90c0c7b30 con 0x7ff90c1e6a90 2023-11-24T06:08:05.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.265+0000 7ff910844700 1 -- 172.21.15.91:0/746818906 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90c1e7e10 con 0x7ff90c1e6a90 2023-11-24T06:08:05.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.266+0000 7ff902ffd700 1 -- 172.21.15.91:0/746818906 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff90401d420 con 0x7ff90c1e6a90 2023-11-24T06:08:05.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.266+0000 7ff902ffd700 1 --2- 172.21.15.91:0/746818906 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff8ec06aed0 0x7ff8ec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.266+0000 7ff902ffd700 1 -- 172.21.15.91:0/746818906 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff90408a980 con 0x7ff90c1e6a90 2023-11-24T06:08:05.276 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.266+0000 7ff90a59c700 1 --2- 172.21.15.91:0/746818906 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff8ec06aed0 0x7ff8ec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.276 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.266+0000 7ff90a59c700 1 --2- 172.21.15.91:0/746818906 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff8ec06aed0 0x7ff8ec06d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc007fd0 tx=0x7ff8fc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:05.276 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.271+0000 7ff902ffd700 1 -- 172.21.15.91:0/746818906 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff90c1e7e10 con 0x7ff90c1e6a90 2023-11-24T06:08:05.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.464+0000 7ff910844700 1 -- 172.21.15.91:0/746818906 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff90c0c7cf0 con 0x7ff90c1e6a90 2023-11-24T06:08:05.469 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.466+0000 7ff902ffd700 1 -- 172.21.15.91:0/746818906 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7ff904055b60 con 0x7ff90c1e6a90 2023-11-24T06:08:05.470 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:05.470 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":20,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:02.415612+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:05.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.468+0000 7ff910844700 1 -- 172.21.15.91:0/746818906 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff8ec06aed0 msgr2=0x7ff8ec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.468+0000 7ff910844700 1 --2- 172.21.15.91:0/746818906 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff8ec06aed0 0x7ff8ec06d390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc007fd0 tx=0x7ff8fc009040 comp rx=0 tx=0).stop 2023-11-24T06:08:05.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.468+0000 7ff910844700 1 -- 172.21.15.91:0/746818906 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff90c1e6a90 msgr2=0x7ff90c144190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.468+0000 7ff910844700 1 --2- 172.21.15.91:0/746818906 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff90c1e6a90 0x7ff90c144190 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff90c0706e0 tx=0x7ff904001d80 comp rx=0 tx=0).stop 2023-11-24T06:08:05.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.468+0000 7ff910844700 1 -- 172.21.15.91:0/746818906 shutdown_connections 2023-11-24T06:08:05.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.468+0000 7ff910844700 1 --2- 172.21.15.91:0/746818906 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff8ec06aed0 0x7ff8ec06d390 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.468+0000 7ff910844700 1 --2- 172.21.15.91:0/746818906 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff90c1e6a90 0x7ff90c144190 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.468+0000 7ff910844700 1 --2- 172.21.15.91:0/746818906 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8f8005210 0x7ff90c0c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.468+0000 7ff910844700 1 --2- 172.21.15.91:0/746818906 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff90c1e80f0 0x7ff90c0c75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.473 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.468+0000 7ff910844700 1 -- 172.21.15.91:0/746818906 >> 172.21.15.91:0/746818906 conn(0x7ff90c0b2740 msgr2=0x7ff90c147680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:05.473 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.468+0000 7ff910844700 1 -- 172.21.15.91:0/746818906 shutdown_connections 2023-11-24T06:08:05.473 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.468+0000 7ff910844700 1 -- 172.21.15.91:0/746818906 wait complete. 2023-11-24T06:08:05.484 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T06:08:05.484 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:08:05.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.600+0000 7f14a8d50700 1 Processor -- start 2023-11-24T06:08:05.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.600+0000 7f14a8d50700 1 -- start start 2023-11-24T06:08:05.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.601+0000 7f14a8d50700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f14a40d06b0 0x7f14a406ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.601+0000 7f14a8d50700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a40b05f0 con 0x7f14a415a440 2023-11-24T06:08:05.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.601+0000 7f14a8d50700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a415f080 con 0x7f14a415a820 2023-11-24T06:08:05.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.601+0000 7f14a8d50700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a40afd50 con 0x7f14a40d06b0 2023-11-24T06:08:05.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.601+0000 7f14a259c700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f14a415a440 0x7f14a40cffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52682/0 (socket says 172.21.15.91:52682) 2023-11-24T06:08:05.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.601+0000 7f14a259c700 1 -- 172.21.15.91:0/3760097262 learned_addr learned my addr 172.21.15.91:0/3760097262 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:05.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.601+0000 7f14a2d9d700 1 --2- 172.21.15.91:0/3760097262 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f14a40d06b0 0x7f14a406ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.601+0000 7f14a159a700 1 -- 172.21.15.91:0/3760097262 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166727799 0 0) 0x7f14a40b05f0 con 0x7f14a415a440 2023-11-24T06:08:05.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.601+0000 7f14a159a700 1 -- 172.21.15.91:0/3760097262 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f148c0035b0 con 0x7f14a415a440 2023-11-24T06:08:05.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.601+0000 7f14a159a700 1 -- 172.21.15.91:0/3760097262 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51741592 0 0) 0x7f14a415f080 con 0x7f14a415a820 2023-11-24T06:08:05.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.601+0000 7f14a159a700 1 -- 172.21.15.91:0/3760097262 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a40b05f0 con 0x7f14a415a820 2023-11-24T06:08:05.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.601+0000 7f14a2d9d700 1 -- 172.21.15.91:0/3760097262 >> v1:172.21.15.94:6789/0 conn(0x7f14a415a820 legacy=0x7f14a406b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.601+0000 7f14a2d9d700 1 -- 172.21.15.91:0/3760097262 >> v1:172.21.15.91:6789/0 conn(0x7f14a415a440 legacy=0x7f14a40cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.601+0000 7f14a2d9d700 1 -- 172.21.15.91:0/3760097262 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a40b0030 con 0x7f14a40d06b0 2023-11-24T06:08:05.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.601+0000 7f14a2d9d700 1 --2- 172.21.15.91:0/3760097262 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f14a40d06b0 0x7f14a406ba50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f149c009dc0 tx=0x7f149c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d46c05b0ae27a93 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:05.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.602+0000 7f14a159a700 1 -- 172.21.15.91:0/3760097262 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c00e070 con 0x7f14a40d06b0 2023-11-24T06:08:05.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.602+0000 7f14a159a700 1 -- 172.21.15.91:0/3760097262 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149c00bb20 con 0x7f14a40d06b0 2023-11-24T06:08:05.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.602+0000 7f14a159a700 1 -- 172.21.15.91:0/3760097262 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c012430 con 0x7f14a40d06b0 2023-11-24T06:08:05.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.602+0000 7f14a8d50700 1 -- 172.21.15.91:0/3760097262 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f14a40d06b0 msgr2=0x7f14a406ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.602+0000 7f14a8d50700 1 --2- 172.21.15.91:0/3760097262 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f14a40d06b0 0x7f14a406ba50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f149c009dc0 tx=0x7f149c00b330 comp rx=0 tx=0).stop 2023-11-24T06:08:05.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.602+0000 7f14a8d50700 1 -- 172.21.15.91:0/3760097262 shutdown_connections 2023-11-24T06:08:05.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.602+0000 7f14a8d50700 1 --2- 172.21.15.91:0/3760097262 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f14a40d06b0 0x7f14a406ba50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.602+0000 7f14a8d50700 1 -- 172.21.15.91:0/3760097262 >> 172.21.15.91:0/3760097262 conn(0x7f14a40b5070 msgr2=0x7f14a40b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:05.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.602+0000 7f14a8d50700 1 -- 172.21.15.91:0/3760097262 shutdown_connections 2023-11-24T06:08:05.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.602+0000 7f14a8d50700 1 -- 172.21.15.91:0/3760097262 wait complete. 2023-11-24T06:08:05.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.604+0000 7f14a8d50700 1 Processor -- start 2023-11-24T06:08:05.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.604+0000 7f14a8d50700 1 -- start start 2023-11-24T06:08:05.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.604+0000 7f14a8d50700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a40d06b0 0x7f14a40ce290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.604+0000 7f14a8d50700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f14a415a440 0x7f14a4095970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.604+0000 7f14a8d50700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f14a4159820 0x7f14a4158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.604+0000 7f14a8d50700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a40ce970 con 0x7f14a4159820 2023-11-24T06:08:05.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.604+0000 7f14a8d50700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a40cec50 con 0x7f14a40d06b0 2023-11-24T06:08:05.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.604+0000 7f14a8d50700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a40cef30 con 0x7f14a415a440 2023-11-24T06:08:05.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.604+0000 7f14a2d9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f14a4159820 0x7f14a4158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.605+0000 7f14a2d9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f14a4159820 0x7f14a4158b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57758/0 (socket says 172.21.15.91:57758) 2023-11-24T06:08:05.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.605+0000 7f14a2d9d700 1 -- 172.21.15.91:0/1726775369 learned_addr learned my addr 172.21.15.91:0/1726775369 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:05.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.605+0000 7f14a1d9b700 1 --2- 172.21.15.91:0/1726775369 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f14a415a440 0x7f14a4095970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.605+0000 7f14a259c700 1 --2- 172.21.15.91:0/1726775369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a40d06b0 0x7f14a40ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.605+0000 7f14a2d9d700 1 -- 172.21.15.91:0/1726775369 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f14a415a440 msgr2=0x7f14a4095970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.605+0000 7f14a2d9d700 1 --2- 172.21.15.91:0/1726775369 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f14a415a440 0x7f14a4095970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.605+0000 7f14a2d9d700 1 -- 172.21.15.91:0/1726775369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a40d06b0 msgr2=0x7f14a40ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.605+0000 7f14a2d9d700 1 --2- 172.21.15.91:0/1726775369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a40d06b0 0x7f14a40ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.605+0000 7f14a2d9d700 1 -- 172.21.15.91:0/1726775369 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149c009a70 con 0x7f14a4159820 2023-11-24T06:08:05.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.605+0000 7f14a259c700 1 --2- 172.21.15.91:0/1726775369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a40d06b0 0x7f14a40ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:05.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.605+0000 7f14a2d9d700 1 --2- 172.21.15.91:0/1726775369 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f14a4159820 0x7f14a4158b10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f149c014040 tx=0x7f149c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:05.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.605+0000 7f1492ffd700 1 -- 172.21.15.91:0/1726775369 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c01b810 con 0x7f14a4159820 2023-11-24T06:08:05.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.605+0000 7f1492ffd700 1 -- 172.21.15.91:0/1726775369 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149c020070 con 0x7f14a4159820 2023-11-24T06:08:05.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.605+0000 7f14a8d50700 1 -- 172.21.15.91:0/1726775369 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a40cf210 con 0x7f14a4159820 2023-11-24T06:08:05.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.605+0000 7f1492ffd700 1 -- 172.21.15.91:0/1726775369 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c01bbb0 con 0x7f14a4159820 2023-11-24T06:08:05.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.605+0000 7f14a8d50700 1 -- 172.21.15.91:0/1726775369 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a41458a0 con 0x7f14a4159820 2023-11-24T06:08:05.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.606+0000 7f14a8d50700 1 -- 172.21.15.91:0/1726775369 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a40cf730 con 0x7f14a4159820 2023-11-24T06:08:05.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.606+0000 7f1492ffd700 1 -- 172.21.15.91:0/1726775369 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f149c01bd10 con 0x7f14a4159820 2023-11-24T06:08:05.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.607+0000 7f1492ffd700 1 --2- 172.21.15.91:0/1726775369 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f148406aed0 0x7f148406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.607+0000 7f1492ffd700 1 -- 172.21.15.91:0/1726775369 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f149c022070 con 0x7f14a4159820 2023-11-24T06:08:05.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.607+0000 7f14a259c700 1 --2- 172.21.15.91:0/1726775369 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f148406aed0 0x7f148406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.607+0000 7f14a259c700 1 --2- 172.21.15.91:0/1726775369 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f148406aed0 0x7f148406d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1494000f60 tx=0x7f1494009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:05.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.611+0000 7f1492ffd700 1 -- 172.21.15.91:0/1726775369 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14a40cf730 con 0x7f14a4159820 2023-11-24T06:08:05.805 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.801+0000 7f14a8d50700 1 -- 172.21.15.91:0/1726775369 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f14a40b1af0 con 0x7f148406aed0 2023-11-24T06:08:05.812 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.809+0000 7f1492ffd700 1 -- 172.21.15.91:0/1726775369 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264604 (secure 0 0 0) 0x7f14a40b1af0 con 0x7f148406aed0 2023-11-24T06:08:05.816 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:05.818 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.812+0000 7f14a8d50700 1 -- 172.21.15.91:0/1726775369 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f148406aed0 msgr2=0x7f148406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.818 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.812+0000 7f14a8d50700 1 --2- 172.21.15.91:0/1726775369 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f148406aed0 0x7f148406d390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1494000f60 tx=0x7f1494009040 comp rx=0 tx=0).stop 2023-11-24T06:08:05.818 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.812+0000 7f14a8d50700 1 -- 172.21.15.91:0/1726775369 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f14a4159820 msgr2=0x7f14a4158b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.818 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.812+0000 7f14a8d50700 1 --2- 172.21.15.91:0/1726775369 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f14a4159820 0x7f14a4158b10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f149c014040 tx=0x7f149c004ad0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.818 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.812+0000 7f14a8d50700 1 -- 172.21.15.91:0/1726775369 shutdown_connections 2023-11-24T06:08:05.819 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.812+0000 7f14a8d50700 1 --2- 172.21.15.91:0/1726775369 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f148406aed0 0x7f148406d390 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.819 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.812+0000 7f14a8d50700 1 --2- 172.21.15.91:0/1726775369 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f14a40d06b0 0x7f14a40ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.819 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.812+0000 7f14a8d50700 1 --2- 172.21.15.91:0/1726775369 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f14a4159820 0x7f14a4158b10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.819 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.812+0000 7f14a8d50700 1 --2- 172.21.15.91:0/1726775369 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f14a415a440 0x7f14a4095970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.819 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.812+0000 7f14a8d50700 1 -- 172.21.15.91:0/1726775369 >> 172.21.15.91:0/1726775369 conn(0x7f14a40b5070 msgr2=0x7f14a4157d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:05.819 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.812+0000 7f14a8d50700 1 -- 172.21.15.91:0/1726775369 shutdown_connections 2023-11-24T06:08:05.819 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.812+0000 7f14a8d50700 1 -- 172.21.15.91:0/1726775369 wait complete. 2023-11-24T06:08:05.819 INFO:teuthology.orchestra.run.smithi091.stderr:dumped all 2023-11-24T06:08:05.828 INFO:teuthology.orchestra.run.smithi091.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-11-24T06:08:05.394031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":97,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218860,"kb_used_data":2692,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755860,"statfs":{"total":773094113280,"available":772870000640,"internally_reserved":0,"allocated":2756608,"data_stored":1693923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.993007"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436794+0000","last_change":"2023-11-24T06:08:01.435066+0000","last_active":"2023-11-24T06:08:01.436794+0000","last_peered":"2023-11-24T06:08:01.436794+0000","last_clean":"2023-11-24T06:08:01.436794+0000","last_became_active":"2023-11-24T06:08:01.434733+0000","last_became_peered":"2023-11-24T06:08:01.434733+0000","last_unstale":"2023-11-24T06:08:01.436794+0000","last_undegraded":"2023-11-24T06:08:01.436794+0000","last_fullsized":"2023-11-24T06:08:01.436794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:22.215190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.430750+0000","last_change":"2023-11-24T06:08:01.428588+0000","last_active":"2023-11-24T06:08:01.430750+0000","last_peered":"2023-11-24T06:08:01.430750+0000","last_clean":"2023-11-24T06:08:01.430750+0000","last_became_active":"2023-11-24T06:08:01.428257+0000","last_became_peered":"2023-11-24T06:08:01.428257+0000","last_unstale":"2023-11-24T06:08:01.430750+0000","last_undegraded":"2023-11-24T06:08:01.430750+0000","last_fullsized":"2023-11-24T06:08:01.430750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:12.434134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.430257+0000","last_change":"2023-11-24T06:08:01.428960+0000","last_active":"2023-11-24T06:08:01.430257+0000","last_peered":"2023-11-24T06:08:01.430257+0000","last_clean":"2023-11-24T06:08:01.430257+0000","last_became_active":"2023-11-24T06:08:01.428600+0000","last_became_peered":"2023-11-24T06:08:01.428600+0000","last_unstale":"2023-11-24T06:08:01.430257+0000","last_undegraded":"2023-11-24T06:08:01.430257+0000","last_fullsized":"2023-11-24T06:08:01.430257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:32.299407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437709+0000","last_change":"2023-11-24T06:08:01.435529+0000","last_active":"2023-11-24T06:08:01.437709+0000","last_peered":"2023-11-24T06:08:01.437709+0000","last_clean":"2023-11-24T06:08:01.437709+0000","last_became_active":"2023-11-24T06:08:01.435128+0000","last_became_peered":"2023-11-24T06:08:01.435128+0000","last_unstale":"2023-11-24T06:08:01.437709+0000","last_undegraded":"2023-11-24T06:08:01.437709+0000","last_fullsized":"2023-11-24T06:08:01.437709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:18.148689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.435401+0000","last_change":"2023-11-24T06:08:01.434208+0000","last_active":"2023-11-24T06:08:01.435401+0000","last_peered":"2023-11-24T06:08:01.435401+0000","last_clean":"2023-11-24T06:08:01.435401+0000","last_became_active":"2023-11-24T06:08:01.433849+0000","last_became_peered":"2023-11-24T06:08:01.433849+0000","last_unstale":"2023-11-24T06:08:01.435401+0000","last_undegraded":"2023-11-24T06:08:01.435401+0000","last_fullsized":"2023-11-24T06:08:01.435401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:37:39.086654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437935+0000","last_change":"2023-11-24T06:08:01.436361+0000","last_active":"2023-11-24T06:08:01.437935+0000","last_peered":"2023-11-24T06:08:01.437935+0000","last_clean":"2023-11-24T06:08:01.437935+0000","last_became_active":"2023-11-24T06:08:01.434166+0000","last_became_peered":"2023-11-24T06:08:01.434166+0000","last_unstale":"2023-11-24T06:08:01.437935+0000","last_undegraded":"2023-11-24T06:08:01.437935+0000","last_fullsized":"2023-11-24T06:08:01.437935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:29.649298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.435270+0000","last_change":"2023-11-24T06:08:01.433541+0000","last_active":"2023-11-24T06:08:01.435270+0000","last_peered":"2023-11-24T06:08:01.435270+0000","last_clean":"2023-11-24T06:08:01.435270+0000","last_became_active":"2023-11-24T06:08:01.433237+0000","last_became_peered":"2023-11-24T06:08:01.433237+0000","last_unstale":"2023-11-24T06:08:01.435270+0000","last_undegraded":"2023-11-24T06:08:01.435270+0000","last_fullsized":"2023-11-24T06:08:01.435270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:21.084798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436565+0000","last_change":"2023-11-24T06:08:01.435678+0000","last_active":"2023-11-24T06:08:01.436565+0000","last_peered":"2023-11-24T06:08:01.436565+0000","last_clean":"2023-11-24T06:08:01.436565+0000","last_became_active":"2023-11-24T06:08:01.435365+0000","last_became_peered":"2023-11-24T06:08:01.435365+0000","last_unstale":"2023-11-24T06:08:01.436565+0000","last_undegraded":"2023-11-24T06:08:01.436565+0000","last_fullsized":"2023-11-24T06:08:01.436565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:29.649298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.439269+0000","last_change":"2023-11-24T06:08:01.438757+0000","last_active":"2023-11-24T06:08:01.439269+0000","last_peered":"2023-11-24T06:08:01.439269+0000","last_clean":"2023-11-24T06:08:01.439269+0000","last_became_active":"2023-11-24T06:08:01.438495+0000","last_became_peered":"2023-11-24T06:08:01.438495+0000","last_unstale":"2023-11-24T06:08:01.439269+0000","last_undegraded":"2023-11-24T06:08:01.439269+0000","last_fullsized":"2023-11-24T06:08:01.439269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:59.812992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437405+0000","last_change":"2023-11-24T06:08:01.435879+0000","last_active":"2023-11-24T06:08:01.437405+0000","last_peered":"2023-11-24T06:08:01.437405+0000","last_clean":"2023-11-24T06:08:01.437405+0000","last_became_active":"2023-11-24T06:08:01.435556+0000","last_became_peered":"2023-11-24T06:08:01.435556+0000","last_unstale":"2023-11-24T06:08:01.437405+0000","last_undegraded":"2023-11-24T06:08:01.437405+0000","last_fullsized":"2023-11-24T06:08:01.437405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:29.649298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437803+0000","last_change":"2023-11-24T06:08:01.436002+0000","last_active":"2023-11-24T06:08:01.437803+0000","last_peered":"2023-11-24T06:08:01.437803+0000","last_clean":"2023-11-24T06:08:01.437803+0000","last_became_active":"2023-11-24T06:08:01.435547+0000","last_became_peered":"2023-11-24T06:08:01.435547+0000","last_unstale":"2023-11-24T06:08:01.437803+0000","last_undegraded":"2023-11-24T06:08:01.437803+0000","last_fullsized":"2023-11-24T06:08:01.437803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:18.148689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.438585+0000","last_change":"2023-11-24T06:08:01.431781+0000","last_active":"2023-11-24T06:08:01.438585+0000","last_peered":"2023-11-24T06:08:01.438585+0000","last_clean":"2023-11-24T06:08:01.438585+0000","last_became_active":"2023-11-24T06:08:01.431494+0000","last_became_peered":"2023-11-24T06:08:01.431494+0000","last_unstale":"2023-11-24T06:08:01.438585+0000","last_undegraded":"2023-11-24T06:08:01.438585+0000","last_fullsized":"2023-11-24T06:08:01.438585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:29.048558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436150+0000","last_change":"2023-11-24T06:08:01.434455+0000","last_active":"2023-11-24T06:08:01.436150+0000","last_peered":"2023-11-24T06:08:01.436150+0000","last_clean":"2023-11-24T06:08:01.436150+0000","last_became_active":"2023-11-24T06:08:01.434201+0000","last_became_peered":"2023-11-24T06:08:01.434201+0000","last_unstale":"2023-11-24T06:08:01.436150+0000","last_undegraded":"2023-11-24T06:08:01.436150+0000","last_fullsized":"2023-11-24T06:08:01.436150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:40.365444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.430133+0000","last_change":"2023-11-24T06:08:01.427810+0000","last_active":"2023-11-24T06:08:01.430133+0000","last_peered":"2023-11-24T06:08:01.430133+0000","last_clean":"2023-11-24T06:08:01.430133+0000","last_became_active":"2023-11-24T06:08:01.427545+0000","last_became_peered":"2023-11-24T06:08:01.427545+0000","last_unstale":"2023-11-24T06:08:01.430133+0000","last_undegraded":"2023-11-24T06:08:01.430133+0000","last_fullsized":"2023-11-24T06:08:01.430133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:59.736889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.432709+0000","last_change":"2023-11-24T06:08:01.431162+0000","last_active":"2023-11-24T06:08:01.432709+0000","last_peered":"2023-11-24T06:08:01.432709+0000","last_clean":"2023-11-24T06:08:01.432709+0000","last_became_active":"2023-11-24T06:08:01.430862+0000","last_became_peered":"2023-11-24T06:08:01.430862+0000","last_unstale":"2023-11-24T06:08:01.432709+0000","last_undegraded":"2023-11-24T06:08:01.432709+0000","last_fullsized":"2023-11-24T06:08:01.432709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:32.299407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437407+0000","last_change":"2023-11-24T06:08:01.434078+0000","last_active":"2023-11-24T06:08:01.437407+0000","last_peered":"2023-11-24T06:08:01.437407+0000","last_clean":"2023-11-24T06:08:01.437407+0000","last_became_active":"2023-11-24T06:08:01.433636+0000","last_became_peered":"2023-11-24T06:08:01.433636+0000","last_unstale":"2023-11-24T06:08:01.437407+0000","last_undegraded":"2023-11-24T06:08:01.437407+0000","last_fullsized":"2023-11-24T06:08:01.437407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:42.556322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.438080+0000","last_change":"2023-11-24T06:08:01.436588+0000","last_active":"2023-11-24T06:08:01.438080+0000","last_peered":"2023-11-24T06:08:01.438080+0000","last_clean":"2023-11-24T06:08:01.438080+0000","last_became_active":"2023-11-24T06:08:01.436193+0000","last_became_peered":"2023-11-24T06:08:01.436193+0000","last_unstale":"2023-11-24T06:08:01.438080+0000","last_undegraded":"2023-11-24T06:08:01.438080+0000","last_fullsized":"2023-11-24T06:08:01.438080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:04.385452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425439+0000","last_change":"2023-11-24T06:08:01.424485+0000","last_active":"2023-11-24T06:08:01.425439+0000","last_peered":"2023-11-24T06:08:01.425439+0000","last_clean":"2023-11-24T06:08:01.425439+0000","last_became_active":"2023-11-24T06:08:01.424234+0000","last_became_peered":"2023-11-24T06:08:01.424234+0000","last_unstale":"2023-11-24T06:08:01.425439+0000","last_undegraded":"2023-11-24T06:08:01.425439+0000","last_fullsized":"2023-11-24T06:08:01.425439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:32.299407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.435713+0000","last_change":"2023-11-24T06:08:01.434319+0000","last_active":"2023-11-24T06:08:01.435713+0000","last_peered":"2023-11-24T06:08:01.435713+0000","last_clean":"2023-11-24T06:08:01.435713+0000","last_became_active":"2023-11-24T06:08:01.433984+0000","last_became_peered":"2023-11-24T06:08:01.433984+0000","last_unstale":"2023-11-24T06:08:01.435713+0000","last_undegraded":"2023-11-24T06:08:01.435713+0000","last_fullsized":"2023-11-24T06:08:01.435713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:52:57.033291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.738808+0000","last_change":"2023-11-24T06:08:00.418990+0000","last_active":"2023-11-24T06:08:01.738808+0000","last_peered":"2023-11-24T06:08:01.738808+0000","last_clean":"2023-11-24T06:08:01.738808+0000","last_became_active":"2023-11-24T06:08:00.418683+0000","last_became_peered":"2023-11-24T06:08:00.418683+0000","last_unstale":"2023-11-24T06:08:01.738808+0000","last_undegraded":"2023-11-24T06:08:01.738808+0000","last_fullsized":"2023-11-24T06:08:01.738808+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:13.152063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433540+0000","last_change":"2023-11-24T06:08:01.431913+0000","last_active":"2023-11-24T06:08:01.433540+0000","last_peered":"2023-11-24T06:08:01.433540+0000","last_clean":"2023-11-24T06:08:01.433540+0000","last_became_active":"2023-11-24T06:08:01.431373+0000","last_became_peered":"2023-11-24T06:08:01.431373+0000","last_unstale":"2023-11-24T06:08:01.433540+0000","last_undegraded":"2023-11-24T06:08:01.433540+0000","last_fullsized":"2023-11-24T06:08:01.433540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:12.434134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.738974+0000","last_change":"2023-11-24T06:08:00.424674+0000","last_active":"2023-11-24T06:08:01.738974+0000","last_peered":"2023-11-24T06:08:01.738974+0000","last_clean":"2023-11-24T06:08:01.738974+0000","last_became_active":"2023-11-24T06:08:00.424282+0000","last_became_peered":"2023-11-24T06:08:00.424282+0000","last_unstale":"2023-11-24T06:08:01.738974+0000","last_undegraded":"2023-11-24T06:08:01.738974+0000","last_fullsized":"2023-11-24T06:08:01.738974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:03.380514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.435944+0000","last_change":"2023-11-24T06:08:01.434659+0000","last_active":"2023-11-24T06:08:01.435944+0000","last_peered":"2023-11-24T06:08:01.435944+0000","last_clean":"2023-11-24T06:08:01.435944+0000","last_became_active":"2023-11-24T06:08:01.434373+0000","last_became_peered":"2023-11-24T06:08:01.434373+0000","last_unstale":"2023-11-24T06:08:01.435944+0000","last_undegraded":"2023-11-24T06:08:01.435944+0000","last_fullsized":"2023-11-24T06:08:01.435944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:19.821244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.736658+0000","last_change":"2023-11-24T06:08:00.426586+0000","last_active":"2023-11-24T06:08:01.736658+0000","last_peered":"2023-11-24T06:08:01.736658+0000","last_clean":"2023-11-24T06:08:01.736658+0000","last_became_active":"2023-11-24T06:08:00.426279+0000","last_became_peered":"2023-11-24T06:08:00.426279+0000","last_unstale":"2023-11-24T06:08:01.736658+0000","last_undegraded":"2023-11-24T06:08:01.736658+0000","last_fullsized":"2023-11-24T06:08:01.736658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:06:57.668726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.441384+0000","last_change":"2023-11-24T06:08:01.439887+0000","last_active":"2023-11-24T06:08:01.441384+0000","last_peered":"2023-11-24T06:08:01.441384+0000","last_clean":"2023-11-24T06:08:01.441384+0000","last_became_active":"2023-11-24T06:08:01.439606+0000","last_became_peered":"2023-11-24T06:08:01.439606+0000","last_unstale":"2023-11-24T06:08:01.441384+0000","last_undegraded":"2023-11-24T06:08:01.441384+0000","last_fullsized":"2023-11-24T06:08:01.441384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:52:57.033291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.737400+0000","last_change":"2023-11-24T06:08:00.423364+0000","last_active":"2023-11-24T06:08:01.737400+0000","last_peered":"2023-11-24T06:08:01.737400+0000","last_clean":"2023-11-24T06:08:01.737400+0000","last_became_active":"2023-11-24T06:08:00.423062+0000","last_became_peered":"2023-11-24T06:08:00.423062+0000","last_unstale":"2023-11-24T06:08:01.737400+0000","last_undegraded":"2023-11-24T06:08:01.737400+0000","last_fullsized":"2023-11-24T06:08:01.737400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:28.644360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.439749+0000","last_change":"2023-11-24T06:08:01.438369+0000","last_active":"2023-11-24T06:08:01.439749+0000","last_peered":"2023-11-24T06:08:01.439749+0000","last_clean":"2023-11-24T06:08:01.439749+0000","last_became_active":"2023-11-24T06:08:01.438112+0000","last_became_peered":"2023-11-24T06:08:01.438112+0000","last_unstale":"2023-11-24T06:08:01.439749+0000","last_undegraded":"2023-11-24T06:08:01.439749+0000","last_fullsized":"2023-11-24T06:08:01.439749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:40.365444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.738620+0000","last_change":"2023-11-24T06:08:00.425844+0000","last_active":"2023-11-24T06:08:01.738620+0000","last_peered":"2023-11-24T06:08:01.738620+0000","last_clean":"2023-11-24T06:08:01.738620+0000","last_became_active":"2023-11-24T06:08:00.425587+0000","last_became_peered":"2023-11-24T06:08:00.425587+0000","last_unstale":"2023-11-24T06:08:01.738620+0000","last_undegraded":"2023-11-24T06:08:01.738620+0000","last_fullsized":"2023-11-24T06:08:01.738620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:29.195643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.434339+0000","last_change":"2023-11-24T06:08:01.427389+0000","last_active":"2023-11-24T06:08:01.434339+0000","last_peered":"2023-11-24T06:08:01.434339+0000","last_clean":"2023-11-24T06:08:01.434339+0000","last_became_active":"2023-11-24T06:08:01.427038+0000","last_became_peered":"2023-11-24T06:08:01.427038+0000","last_unstale":"2023-11-24T06:08:01.434339+0000","last_undegraded":"2023-11-24T06:08:01.434339+0000","last_fullsized":"2023-11-24T06:08:01.434339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:19.821244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.735862+0000","last_change":"2023-11-24T06:08:00.411792+0000","last_active":"2023-11-24T06:08:01.735862+0000","last_peered":"2023-11-24T06:08:01.735862+0000","last_clean":"2023-11-24T06:08:01.735862+0000","last_became_active":"2023-11-24T06:08:00.411486+0000","last_became_peered":"2023-11-24T06:08:00.411486+0000","last_unstale":"2023-11-24T06:08:01.735862+0000","last_undegraded":"2023-11-24T06:08:01.735862+0000","last_fullsized":"2023-11-24T06:08:01.735862+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:41.551384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.445646+0000","last_change":"2023-11-24T06:08:01.438231+0000","last_active":"2023-11-24T06:08:01.445646+0000","last_peered":"2023-11-24T06:08:01.445646+0000","last_clean":"2023-11-24T06:08:01.445646+0000","last_became_active":"2023-11-24T06:08:01.437944+0000","last_became_peered":"2023-11-24T06:08:01.437944+0000","last_unstale":"2023-11-24T06:08:01.445646+0000","last_undegraded":"2023-11-24T06:08:01.445646+0000","last_fullsized":"2023-11-24T06:08:01.445646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:16.537475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.421698+0000","last_change":"2023-11-24T06:08:00.416967+0000","last_active":"2023-11-24T06:08:01.421698+0000","last_peered":"2023-11-24T06:08:01.421698+0000","last_clean":"2023-11-24T06:08:01.421698+0000","last_became_active":"2023-11-24T06:08:00.416532+0000","last_became_peered":"2023-11-24T06:08:00.416532+0000","last_unstale":"2023-11-24T06:08:01.421698+0000","last_undegraded":"2023-11-24T06:08:01.421698+0000","last_fullsized":"2023-11-24T06:08:01.421698+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:20.079860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433288+0000","last_change":"2023-11-24T06:08:01.429802+0000","last_active":"2023-11-24T06:08:01.433288+0000","last_peered":"2023-11-24T06:08:01.433288+0000","last_clean":"2023-11-24T06:08:01.433288+0000","last_became_active":"2023-11-24T06:08:01.429578+0000","last_became_peered":"2023-11-24T06:08:01.429578+0000","last_unstale":"2023-11-24T06:08:01.433288+0000","last_undegraded":"2023-11-24T06:08:01.433288+0000","last_fullsized":"2023-11-24T06:08:01.433288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:19.821244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.427448+0000","last_change":"2023-11-24T06:08:00.422030+0000","last_active":"2023-11-24T06:08:01.427448+0000","last_peered":"2023-11-24T06:08:01.427448+0000","last_clean":"2023-11-24T06:08:01.427448+0000","last_became_active":"2023-11-24T06:08:00.421740+0000","last_became_peered":"2023-11-24T06:08:00.421740+0000","last_unstale":"2023-11-24T06:08:01.427448+0000","last_undegraded":"2023-11-24T06:08:01.427448+0000","last_fullsized":"2023-11-24T06:08:01.427448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:58.808054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436372+0000","last_change":"2023-11-24T06:08:01.433496+0000","last_active":"2023-11-24T06:08:01.436372+0000","last_peered":"2023-11-24T06:08:01.436372+0000","last_clean":"2023-11-24T06:08:01.436372+0000","last_became_active":"2023-11-24T06:08:01.433238+0000","last_became_peered":"2023-11-24T06:08:01.433238+0000","last_unstale":"2023-11-24T06:08:01.436372+0000","last_undegraded":"2023-11-24T06:08:01.436372+0000","last_fullsized":"2023-11-24T06:08:01.436372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:14.157001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425023+0000","last_change":"2023-11-24T06:08:00.422595+0000","last_active":"2023-11-24T06:08:01.425023+0000","last_peered":"2023-11-24T06:08:01.425023+0000","last_clean":"2023-11-24T06:08:01.425023+0000","last_became_active":"2023-11-24T06:08:00.422365+0000","last_became_peered":"2023-11-24T06:08:00.422365+0000","last_unstale":"2023-11-24T06:08:01.425023+0000","last_undegraded":"2023-11-24T06:08:01.425023+0000","last_fullsized":"2023-11-24T06:08:01.425023+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:39.360506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437390+0000","last_change":"2023-11-24T06:08:01.434890+0000","last_active":"2023-11-24T06:08:01.437390+0000","last_peered":"2023-11-24T06:08:01.437390+0000","last_clean":"2023-11-24T06:08:01.437390+0000","last_became_active":"2023-11-24T06:08:01.434706+0000","last_became_peered":"2023-11-24T06:08:01.434706+0000","last_unstale":"2023-11-24T06:08:01.437390+0000","last_undegraded":"2023-11-24T06:08:01.437390+0000","last_fullsized":"2023-11-24T06:08:01.437390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:21.084798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425886+0000","last_change":"2023-11-24T06:08:00.423297+0000","last_active":"2023-11-24T06:08:01.425886+0000","last_peered":"2023-11-24T06:08:01.425886+0000","last_clean":"2023-11-24T06:08:01.425886+0000","last_became_active":"2023-11-24T06:08:00.423028+0000","last_became_peered":"2023-11-24T06:08:00.423028+0000","last_unstale":"2023-11-24T06:08:01.425886+0000","last_undegraded":"2023-11-24T06:08:01.425886+0000","last_fullsized":"2023-11-24T06:08:01.425886+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:37:38.081716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.440137+0000","last_change":"2023-11-24T06:08:01.437725+0000","last_active":"2023-11-24T06:08:01.440137+0000","last_peered":"2023-11-24T06:08:01.440137+0000","last_clean":"2023-11-24T06:08:01.440137+0000","last_became_active":"2023-11-24T06:08:01.437464+0000","last_became_peered":"2023-11-24T06:08:01.437464+0000","last_unstale":"2023-11-24T06:08:01.440137+0000","last_undegraded":"2023-11-24T06:08:01.440137+0000","last_fullsized":"2023-11-24T06:08:01.440137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:21:46.421251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.737950+0000","last_change":"2023-11-24T06:08:00.421041+0000","last_active":"2023-11-24T06:08:01.737950+0000","last_peered":"2023-11-24T06:08:01.737950+0000","last_clean":"2023-11-24T06:08:01.737950+0000","last_became_active":"2023-11-24T06:08:00.420677+0000","last_became_peered":"2023-11-24T06:08:00.420677+0000","last_unstale":"2023-11-24T06:08:01.737950+0000","last_undegraded":"2023-11-24T06:08:01.737950+0000","last_fullsized":"2023-11-24T06:08:01.737950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:29.195643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.439962+0000","last_change":"2023-11-24T06:08:01.437971+0000","last_active":"2023-11-24T06:08:01.439962+0000","last_peered":"2023-11-24T06:08:01.439962+0000","last_clean":"2023-11-24T06:08:01.439962+0000","last_became_active":"2023-11-24T06:08:01.437640+0000","last_became_peered":"2023-11-24T06:08:01.437640+0000","last_unstale":"2023-11-24T06:08:01.439962+0000","last_undegraded":"2023-11-24T06:08:01.439962+0000","last_fullsized":"2023-11-24T06:08:01.439962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:59.736889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.741721+0000","last_change":"2023-11-24T06:08:00.418490+0000","last_active":"2023-11-24T06:08:01.741721+0000","last_peered":"2023-11-24T06:08:01.741721+0000","last_clean":"2023-11-24T06:08:01.741721+0000","last_became_active":"2023-11-24T06:08:00.418200+0000","last_became_peered":"2023-11-24T06:08:00.418200+0000","last_unstale":"2023-11-24T06:08:01.741721+0000","last_undegraded":"2023-11-24T06:08:01.741721+0000","last_fullsized":"2023-11-24T06:08:01.741721+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:03.380514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.442296+0000","last_change":"2023-11-24T06:08:01.439461+0000","last_active":"2023-11-24T06:08:01.442296+0000","last_peered":"2023-11-24T06:08:01.442296+0000","last_clean":"2023-11-24T06:08:01.442296+0000","last_became_active":"2023-11-24T06:08:01.439210+0000","last_became_peered":"2023-11-24T06:08:01.439210+0000","last_unstale":"2023-11-24T06:08:01.442296+0000","last_undegraded":"2023-11-24T06:08:01.442296+0000","last_fullsized":"2023-11-24T06:08:01.442296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:28.564398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.736609+0000","last_change":"2023-11-24T06:08:00.423172+0000","last_active":"2023-11-24T06:08:01.736609+0000","last_peered":"2023-11-24T06:08:01.736609+0000","last_clean":"2023-11-24T06:08:01.736609+0000","last_became_active":"2023-11-24T06:08:00.422899+0000","last_became_peered":"2023-11-24T06:08:00.422899+0000","last_unstale":"2023-11-24T06:08:01.736609+0000","last_undegraded":"2023-11-24T06:08:01.736609+0000","last_fullsized":"2023-11-24T06:08:01.736609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:17.143751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436314+0000","last_change":"2023-11-24T06:08:01.435150+0000","last_active":"2023-11-24T06:08:01.436314+0000","last_peered":"2023-11-24T06:08:01.436314+0000","last_clean":"2023-11-24T06:08:01.436314+0000","last_became_active":"2023-11-24T06:08:01.434861+0000","last_became_peered":"2023-11-24T06:08:01.434861+0000","last_unstale":"2023-11-24T06:08:01.436314+0000","last_undegraded":"2023-11-24T06:08:01.436314+0000","last_fullsized":"2023-11-24T06:08:01.436314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:32.624231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.429337+0000","last_change":"2023-11-24T06:08:00.422660+0000","last_active":"2023-11-24T06:08:01.429337+0000","last_peered":"2023-11-24T06:08:01.429337+0000","last_clean":"2023-11-24T06:08:01.429337+0000","last_became_active":"2023-11-24T06:08:00.421922+0000","last_became_peered":"2023-11-24T06:08:00.421922+0000","last_unstale":"2023-11-24T06:08:01.429337+0000","last_undegraded":"2023-11-24T06:08:01.429337+0000","last_fullsized":"2023-11-24T06:08:01.429337+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:27.559460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.443240+0000","last_change":"2023-11-24T06:08:01.442581+0000","last_active":"2023-11-24T06:08:01.443240+0000","last_peered":"2023-11-24T06:08:01.443240+0000","last_clean":"2023-11-24T06:08:01.443240+0000","last_became_active":"2023-11-24T06:08:01.442145+0000","last_became_peered":"2023-11-24T06:08:01.442145+0000","last_unstale":"2023-11-24T06:08:01.443240+0000","last_undegraded":"2023-11-24T06:08:01.443240+0000","last_fullsized":"2023-11-24T06:08:01.443240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:04.385452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.419830+0000","last_change":"2023-11-24T06:08:00.413903+0000","last_active":"2023-11-24T06:08:01.419830+0000","last_peered":"2023-11-24T06:08:01.419830+0000","last_clean":"2023-11-24T06:08:01.419830+0000","last_became_active":"2023-11-24T06:08:00.413581+0000","last_became_peered":"2023-11-24T06:08:00.413581+0000","last_unstale":"2023-11-24T06:08:01.419830+0000","last_undegraded":"2023-11-24T06:08:01.419830+0000","last_fullsized":"2023-11-24T06:08:01.419830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:15.532537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433217+0000","last_change":"2023-11-24T06:08:01.429649+0000","last_active":"2023-11-24T06:08:01.433217+0000","last_peered":"2023-11-24T06:08:01.433217+0000","last_clean":"2023-11-24T06:08:01.433217+0000","last_became_active":"2023-11-24T06:08:01.429189+0000","last_became_peered":"2023-11-24T06:08:01.429189+0000","last_unstale":"2023-11-24T06:08:01.433217+0000","last_undegraded":"2023-11-24T06:08:01.433217+0000","last_fullsized":"2023-11-24T06:08:01.433217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:21.084798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.739985+0000","last_change":"2023-11-24T06:08:00.417220+0000","last_active":"2023-11-24T06:08:01.739985+0000","last_peered":"2023-11-24T06:08:01.739985+0000","last_clean":"2023-11-24T06:08:01.739985+0000","last_became_active":"2023-11-24T06:08:00.416947+0000","last_became_peered":"2023-11-24T06:08:00.416947+0000","last_unstale":"2023-11-24T06:08:01.739985+0000","last_undegraded":"2023-11-24T06:08:01.739985+0000","last_fullsized":"2023-11-24T06:08:01.739985+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:18.816306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.442913+0000","last_change":"2023-11-24T06:08:01.442182+0000","last_active":"2023-11-24T06:08:01.442913+0000","last_peered":"2023-11-24T06:08:01.442913+0000","last_clean":"2023-11-24T06:08:01.442913+0000","last_became_active":"2023-11-24T06:08:01.441895+0000","last_became_peered":"2023-11-24T06:08:01.441895+0000","last_unstale":"2023-11-24T06:08:01.442913+0000","last_undegraded":"2023-11-24T06:08:01.442913+0000","last_fullsized":"2023-11-24T06:08:01.442913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:47:16.508162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.738716+0000","last_change":"2023-11-24T06:08:00.420431+0000","last_active":"2023-11-24T06:08:01.738716+0000","last_peered":"2023-11-24T06:08:01.738716+0000","last_clean":"2023-11-24T06:08:01.738716+0000","last_became_active":"2023-11-24T06:08:00.420108+0000","last_became_peered":"2023-11-24T06:08:00.420108+0000","last_unstale":"2023-11-24T06:08:01.738716+0000","last_undegraded":"2023-11-24T06:08:01.738716+0000","last_fullsized":"2023-11-24T06:08:01.738716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:11.429196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.442887+0000","last_change":"2023-11-24T06:08:01.441968+0000","last_active":"2023-11-24T06:08:01.442887+0000","last_peered":"2023-11-24T06:08:01.442887+0000","last_clean":"2023-11-24T06:08:01.442887+0000","last_became_active":"2023-11-24T06:08:01.440920+0000","last_became_peered":"2023-11-24T06:08:01.440920+0000","last_unstale":"2023-11-24T06:08:01.442887+0000","last_undegraded":"2023-11-24T06:08:01.442887+0000","last_fullsized":"2023-11-24T06:08:01.442887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:40:20.503614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425271+0000","last_change":"2023-11-24T06:08:00.423440+0000","last_active":"2023-11-24T06:08:01.425271+0000","last_peered":"2023-11-24T06:08:01.425271+0000","last_clean":"2023-11-24T06:08:01.425271+0000","last_became_active":"2023-11-24T06:08:00.423175+0000","last_became_peered":"2023-11-24T06:08:00.423175+0000","last_unstale":"2023-11-24T06:08:01.425271+0000","last_undegraded":"2023-11-24T06:08:01.425271+0000","last_fullsized":"2023-11-24T06:08:01.425271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:39.360506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437509+0000","last_change":"2023-11-24T06:08:01.435280+0000","last_active":"2023-11-24T06:08:01.437509+0000","last_peered":"2023-11-24T06:08:01.437509+0000","last_clean":"2023-11-24T06:08:01.437509+0000","last_became_active":"2023-11-24T06:08:01.434963+0000","last_became_peered":"2023-11-24T06:08:01.434963+0000","last_unstale":"2023-11-24T06:08:01.437509+0000","last_undegraded":"2023-11-24T06:08:01.437509+0000","last_fullsized":"2023-11-24T06:08:01.437509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:16.537475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.428526+0000","last_change":"2023-11-24T06:08:00.420129+0000","last_active":"2023-11-24T06:08:01.428526+0000","last_peered":"2023-11-24T06:08:01.428526+0000","last_clean":"2023-11-24T06:08:01.428526+0000","last_became_active":"2023-11-24T06:08:00.418612+0000","last_became_peered":"2023-11-24T06:08:00.418612+0000","last_unstale":"2023-11-24T06:08:01.428526+0000","last_undegraded":"2023-11-24T06:08:01.428526+0000","last_fullsized":"2023-11-24T06:08:01.428526+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:40:19.498676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.439553+0000","last_change":"2023-11-24T06:08:01.430171+0000","last_active":"2023-11-24T06:08:01.439553+0000","last_peered":"2023-11-24T06:08:01.439553+0000","last_clean":"2023-11-24T06:08:01.439553+0000","last_became_active":"2023-11-24T06:08:01.428562+0000","last_became_peered":"2023-11-24T06:08:01.428562+0000","last_unstale":"2023-11-24T06:08:01.439553+0000","last_undegraded":"2023-11-24T06:08:01.439553+0000","last_fullsized":"2023-11-24T06:08:01.439553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:32.624231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436350+0000","last_change":"2023-11-24T06:08:01.433086+0000","last_active":"2023-11-24T06:08:01.436350+0000","last_peered":"2023-11-24T06:08:01.436350+0000","last_clean":"2023-11-24T06:08:01.436350+0000","last_became_active":"2023-11-24T06:08:01.432747+0000","last_became_peered":"2023-11-24T06:08:01.432747+0000","last_unstale":"2023-11-24T06:08:01.436350+0000","last_undegraded":"2023-11-24T06:08:01.436350+0000","last_fullsized":"2023-11-24T06:08:01.436350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:42.556322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.417666+0000","last_change":"2023-11-24T06:08:00.416138+0000","last_active":"2023-11-24T06:08:01.417666+0000","last_peered":"2023-11-24T06:08:01.417666+0000","last_clean":"2023-11-24T06:08:01.417666+0000","last_became_active":"2023-11-24T06:08:00.415909+0000","last_became_peered":"2023-11-24T06:08:00.415909+0000","last_unstale":"2023-11-24T06:08:01.417666+0000","last_undegraded":"2023-11-24T06:08:01.417666+0000","last_fullsized":"2023-11-24T06:08:01.417666+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:31.619293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.445658+0000","last_change":"2023-11-24T06:08:01.438377+0000","last_active":"2023-11-24T06:08:01.445658+0000","last_peered":"2023-11-24T06:08:01.445658+0000","last_clean":"2023-11-24T06:08:01.445658+0000","last_became_active":"2023-11-24T06:08:01.438142+0000","last_became_peered":"2023-11-24T06:08:01.438142+0000","last_unstale":"2023-11-24T06:08:01.445658+0000","last_undegraded":"2023-11-24T06:08:01.445658+0000","last_fullsized":"2023-11-24T06:08:01.445658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:37:39.086654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.423775+0000","last_change":"2023-11-24T06:08:00.419708+0000","last_active":"2023-11-24T06:08:01.423775+0000","last_peered":"2023-11-24T06:08:01.423775+0000","last_clean":"2023-11-24T06:08:01.423775+0000","last_became_active":"2023-11-24T06:08:00.419444+0000","last_became_peered":"2023-11-24T06:08:00.419444+0000","last_unstale":"2023-11-24T06:08:01.423775+0000","last_undegraded":"2023-11-24T06:08:01.423775+0000","last_fullsized":"2023-11-24T06:08:01.423775+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:13.152063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-11-24T06:08:00.417748+0000","last_change":"2023-11-24T06:07:49.463719+0000","last_active":"2023-11-24T06:08:00.417748+0000","last_peered":"2023-11-24T06:08:00.417748+0000","last_clean":"2023-11-24T06:08:00.417748+0000","last_became_active":"2023-11-24T06:07:49.462736+0000","last_became_peered":"2023-11-24T06:07:49.462736+0000","last_unstale":"2023-11-24T06:08:00.417748+0000","last_undegraded":"2023-11-24T06:08:00.417748+0000","last_fullsized":"2023-11-24T06:08:00.417748+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_clean_scrub_stamp":"2023-11-24T06:07:47.860604+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:16.508770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.445621+0000","last_change":"2023-11-24T06:08:01.438231+0000","last_active":"2023-11-24T06:08:01.445621+0000","last_peered":"2023-11-24T06:08:01.445621+0000","last_clean":"2023-11-24T06:08:01.445621+0000","last_became_active":"2023-11-24T06:08:01.437970+0000","last_became_peered":"2023-11-24T06:08:01.437970+0000","last_unstale":"2023-11-24T06:08:01.445621+0000","last_undegraded":"2023-11-24T06:08:01.445621+0000","last_fullsized":"2023-11-24T06:08:01.445621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:42.556322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425496+0000","last_change":"2023-11-24T06:08:00.423316+0000","last_active":"2023-11-24T06:08:01.425496+0000","last_peered":"2023-11-24T06:08:01.425496+0000","last_clean":"2023-11-24T06:08:01.425496+0000","last_became_active":"2023-11-24T06:08:00.423046+0000","last_became_peered":"2023-11-24T06:08:00.423046+0000","last_unstale":"2023-11-24T06:08:01.425496+0000","last_undegraded":"2023-11-24T06:08:01.425496+0000","last_fullsized":"2023-11-24T06:08:01.425496+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:17.143751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.429067+0000","last_change":"2023-11-24T06:08:00.424803+0000","last_active":"2023-11-24T06:08:01.429067+0000","last_peered":"2023-11-24T06:08:01.429067+0000","last_clean":"2023-11-24T06:08:01.429067+0000","last_became_active":"2023-11-24T06:08:00.424473+0000","last_became_peered":"2023-11-24T06:08:00.424473+0000","last_unstale":"2023-11-24T06:08:01.429067+0000","last_undegraded":"2023-11-24T06:08:01.429067+0000","last_fullsized":"2023-11-24T06:08:01.429067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:00.828006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.432957+0000","last_change":"2023-11-24T06:08:01.432093+0000","last_active":"2023-11-24T06:08:01.432957+0000","last_peered":"2023-11-24T06:08:01.432957+0000","last_clean":"2023-11-24T06:08:01.432957+0000","last_became_active":"2023-11-24T06:08:01.431795+0000","last_became_peered":"2023-11-24T06:08:01.431795+0000","last_unstale":"2023-11-24T06:08:01.432957+0000","last_undegraded":"2023-11-24T06:08:01.432957+0000","last_fullsized":"2023-11-24T06:08:01.432957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:37:39.086654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425234+0000","last_change":"2023-11-24T06:08:00.416062+0000","last_active":"2023-11-24T06:08:01.425234+0000","last_peered":"2023-11-24T06:08:01.425234+0000","last_clean":"2023-11-24T06:08:01.425234+0000","last_became_active":"2023-11-24T06:08:00.415783+0000","last_became_peered":"2023-11-24T06:08:00.415783+0000","last_unstale":"2023-11-24T06:08:01.425234+0000","last_undegraded":"2023-11-24T06:08:01.425234+0000","last_fullsized":"2023-11-24T06:08:01.425234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:03.380514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.445642+0000","last_change":"2023-11-24T06:08:01.437911+0000","last_active":"2023-11-24T06:08:01.445642+0000","last_peered":"2023-11-24T06:08:01.445642+0000","last_clean":"2023-11-24T06:08:01.445642+0000","last_became_active":"2023-11-24T06:08:01.437544+0000","last_became_peered":"2023-11-24T06:08:01.437544+0000","last_unstale":"2023-11-24T06:08:01.445642+0000","last_undegraded":"2023-11-24T06:08:01.445642+0000","last_fullsized":"2023-11-24T06:08:01.445642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:42.556322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.736546+0000","last_change":"2023-11-24T06:08:00.421447+0000","last_active":"2023-11-24T06:08:01.736546+0000","last_peered":"2023-11-24T06:08:01.736546+0000","last_clean":"2023-11-24T06:08:01.736546+0000","last_became_active":"2023-11-24T06:08:00.421145+0000","last_became_peered":"2023-11-24T06:08:00.421145+0000","last_unstale":"2023-11-24T06:08:01.736546+0000","last_undegraded":"2023-11-24T06:08:01.736546+0000","last_fullsized":"2023-11-24T06:08:01.736546+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:29.195643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.430086+0000","last_change":"2023-11-24T06:08:01.427774+0000","last_active":"2023-11-24T06:08:01.430086+0000","last_peered":"2023-11-24T06:08:01.430086+0000","last_clean":"2023-11-24T06:08:01.430086+0000","last_became_active":"2023-11-24T06:08:01.427435+0000","last_became_peered":"2023-11-24T06:08:01.427435+0000","last_unstale":"2023-11-24T06:08:01.430086+0000","last_undegraded":"2023-11-24T06:08:01.430086+0000","last_fullsized":"2023-11-24T06:08:01.430086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:12.434134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425252+0000","last_change":"2023-11-24T06:08:00.424213+0000","last_active":"2023-11-24T06:08:01.425252+0000","last_peered":"2023-11-24T06:08:01.425252+0000","last_clean":"2023-11-24T06:08:01.425252+0000","last_became_active":"2023-11-24T06:08:00.423943+0000","last_became_peered":"2023-11-24T06:08:00.423943+0000","last_unstale":"2023-11-24T06:08:01.425252+0000","last_undegraded":"2023-11-24T06:08:01.425252+0000","last_fullsized":"2023-11-24T06:08:01.425252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:15.532537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.422192+0000","last_change":"2023-11-24T06:08:01.420581+0000","last_active":"2023-11-24T06:08:01.422192+0000","last_peered":"2023-11-24T06:08:01.422192+0000","last_clean":"2023-11-24T06:08:01.422192+0000","last_became_active":"2023-11-24T06:08:01.420103+0000","last_became_peered":"2023-11-24T06:08:01.420103+0000","last_unstale":"2023-11-24T06:08:01.422192+0000","last_undegraded":"2023-11-24T06:08:01.422192+0000","last_fullsized":"2023-11-24T06:08:01.422192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:21:46.421251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.423958+0000","last_change":"2023-11-24T06:08:00.420593+0000","last_active":"2023-11-24T06:08:01.423958+0000","last_peered":"2023-11-24T06:08:01.423958+0000","last_clean":"2023-11-24T06:08:01.423958+0000","last_became_active":"2023-11-24T06:08:00.420286+0000","last_became_peered":"2023-11-24T06:08:00.420286+0000","last_unstale":"2023-11-24T06:08:01.423958+0000","last_undegraded":"2023-11-24T06:08:01.423958+0000","last_fullsized":"2023-11-24T06:08:01.423958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:28.644360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.432944+0000","last_change":"2023-11-24T06:08:01.427055+0000","last_active":"2023-11-24T06:08:01.432944+0000","last_peered":"2023-11-24T06:08:01.432944+0000","last_clean":"2023-11-24T06:08:01.432944+0000","last_became_active":"2023-11-24T06:08:01.426823+0000","last_became_peered":"2023-11-24T06:08:01.426823+0000","last_unstale":"2023-11-24T06:08:01.432944+0000","last_undegraded":"2023-11-24T06:08:01.432944+0000","last_fullsized":"2023-11-24T06:08:01.432944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:32.299407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425355+0000","last_change":"2023-11-24T06:08:00.421597+0000","last_active":"2023-11-24T06:08:01.425355+0000","last_peered":"2023-11-24T06:08:01.425355+0000","last_clean":"2023-11-24T06:08:01.425355+0000","last_became_active":"2023-11-24T06:08:00.421316+0000","last_became_peered":"2023-11-24T06:08:00.421316+0000","last_unstale":"2023-11-24T06:08:01.425355+0000","last_undegraded":"2023-11-24T06:08:01.425355+0000","last_fullsized":"2023-11-24T06:08:01.425355+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:17.143751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433264+0000","last_change":"2023-11-24T06:08:01.430666+0000","last_active":"2023-11-24T06:08:01.433264+0000","last_peered":"2023-11-24T06:08:01.433264+0000","last_clean":"2023-11-24T06:08:01.433264+0000","last_became_active":"2023-11-24T06:08:01.430401+0000","last_became_peered":"2023-11-24T06:08:01.430401+0000","last_unstale":"2023-11-24T06:08:01.433264+0000","last_undegraded":"2023-11-24T06:08:01.433264+0000","last_fullsized":"2023-11-24T06:08:01.433264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:14.157001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.736248+0000","last_change":"2023-11-24T06:08:00.424291+0000","last_active":"2023-11-24T06:08:01.736248+0000","last_peered":"2023-11-24T06:08:01.736248+0000","last_clean":"2023-11-24T06:08:01.736248+0000","last_became_active":"2023-11-24T06:08:00.422721+0000","last_became_peered":"2023-11-24T06:08:00.422721+0000","last_unstale":"2023-11-24T06:08:01.736248+0000","last_undegraded":"2023-11-24T06:08:01.736248+0000","last_fullsized":"2023-11-24T06:08:01.736248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:29.195643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433345+0000","last_change":"2023-11-24T06:08:01.424141+0000","last_active":"2023-11-24T06:08:01.433345+0000","last_peered":"2023-11-24T06:08:01.433345+0000","last_clean":"2023-11-24T06:08:01.433345+0000","last_became_active":"2023-11-24T06:08:01.423837+0000","last_became_peered":"2023-11-24T06:08:01.423837+0000","last_unstale":"2023-11-24T06:08:01.433345+0000","last_undegraded":"2023-11-24T06:08:01.433345+0000","last_fullsized":"2023-11-24T06:08:01.433345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:12.434134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.417729+0000","last_change":"2023-11-24T06:08:00.420556+0000","last_active":"2023-11-24T06:08:01.417729+0000","last_peered":"2023-11-24T06:08:01.417729+0000","last_clean":"2023-11-24T06:08:01.417729+0000","last_became_active":"2023-11-24T06:08:00.420250+0000","last_became_peered":"2023-11-24T06:08:00.420250+0000","last_unstale":"2023-11-24T06:08:01.417729+0000","last_undegraded":"2023-11-24T06:08:01.417729+0000","last_fullsized":"2023-11-24T06:08:01.417729+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:11.429196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.442315+0000","last_change":"2023-11-24T06:08:01.440217+0000","last_active":"2023-11-24T06:08:01.442315+0000","last_peered":"2023-11-24T06:08:01.442315+0000","last_clean":"2023-11-24T06:08:01.442315+0000","last_became_active":"2023-11-24T06:08:01.439942+0000","last_became_peered":"2023-11-24T06:08:01.439942+0000","last_unstale":"2023-11-24T06:08:01.442315+0000","last_undegraded":"2023-11-24T06:08:01.442315+0000","last_fullsized":"2023-11-24T06:08:01.442315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:52.422428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.738694+0000","last_change":"2023-11-24T06:08:00.413735+0000","last_active":"2023-11-24T06:08:01.738694+0000","last_peered":"2023-11-24T06:08:01.738694+0000","last_clean":"2023-11-24T06:08:01.738694+0000","last_became_active":"2023-11-24T06:08:00.413375+0000","last_became_peered":"2023-11-24T06:08:00.413375+0000","last_unstale":"2023-11-24T06:08:01.738694+0000","last_undegraded":"2023-11-24T06:08:01.738694+0000","last_fullsized":"2023-11-24T06:08:01.738694+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:20.079860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437726+0000","last_change":"2023-11-24T06:08:01.435772+0000","last_active":"2023-11-24T06:08:01.437726+0000","last_peered":"2023-11-24T06:08:01.437726+0000","last_clean":"2023-11-24T06:08:01.437726+0000","last_became_active":"2023-11-24T06:08:01.435495+0000","last_became_peered":"2023-11-24T06:08:01.435495+0000","last_unstale":"2023-11-24T06:08:01.437726+0000","last_undegraded":"2023-11-24T06:08:01.437726+0000","last_fullsized":"2023-11-24T06:08:01.437726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:30.200581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.738729+0000","last_change":"2023-11-24T06:08:00.415129+0000","last_active":"2023-11-24T06:08:01.738729+0000","last_peered":"2023-11-24T06:08:01.738729+0000","last_clean":"2023-11-24T06:08:01.738729+0000","last_became_active":"2023-11-24T06:08:00.414888+0000","last_became_peered":"2023-11-24T06:08:00.414888+0000","last_unstale":"2023-11-24T06:08:01.738729+0000","last_undegraded":"2023-11-24T06:08:01.738729+0000","last_fullsized":"2023-11-24T06:08:01.738729+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:41.551384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.443231+0000","last_change":"2023-11-24T06:08:01.442379+0000","last_active":"2023-11-24T06:08:01.443231+0000","last_peered":"2023-11-24T06:08:01.443231+0000","last_clean":"2023-11-24T06:08:01.443231+0000","last_became_active":"2023-11-24T06:08:01.442109+0000","last_became_peered":"2023-11-24T06:08:01.442109+0000","last_unstale":"2023-11-24T06:08:01.443231+0000","last_undegraded":"2023-11-24T06:08:01.443231+0000","last_fullsized":"2023-11-24T06:08:01.443231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:40.365444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433238+0000","last_change":"2023-11-24T06:08:01.429866+0000","last_active":"2023-11-24T06:08:01.433238+0000","last_peered":"2023-11-24T06:08:01.433238+0000","last_clean":"2023-11-24T06:08:01.433238+0000","last_became_active":"2023-11-24T06:08:01.429446+0000","last_became_peered":"2023-11-24T06:08:01.429446+0000","last_unstale":"2023-11-24T06:08:01.433238+0000","last_undegraded":"2023-11-24T06:08:01.433238+0000","last_fullsized":"2023-11-24T06:08:01.433238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:14.157001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436208+0000","last_change":"2023-11-24T06:08:01.435709+0000","last_active":"2023-11-24T06:08:01.436208+0000","last_peered":"2023-11-24T06:08:01.436208+0000","last_clean":"2023-11-24T06:08:01.436208+0000","last_became_active":"2023-11-24T06:08:01.435488+0000","last_became_peered":"2023-11-24T06:08:01.435488+0000","last_unstale":"2023-11-24T06:08:01.436208+0000","last_undegraded":"2023-11-24T06:08:01.436208+0000","last_fullsized":"2023-11-24T06:08:01.436208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:14.157001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437910+0000","last_change":"2023-11-24T06:08:01.434372+0000","last_active":"2023-11-24T06:08:01.437910+0000","last_peered":"2023-11-24T06:08:01.437910+0000","last_clean":"2023-11-24T06:08:01.437910+0000","last_became_active":"2023-11-24T06:08:01.433776+0000","last_became_peered":"2023-11-24T06:08:01.433776+0000","last_unstale":"2023-11-24T06:08:01.437910+0000","last_undegraded":"2023-11-24T06:08:01.437910+0000","last_fullsized":"2023-11-24T06:08:01.437910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:12.434134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.435884+0000","last_change":"2023-11-24T06:08:01.433754+0000","last_active":"2023-11-24T06:08:01.435884+0000","last_peered":"2023-11-24T06:08:01.435884+0000","last_clean":"2023-11-24T06:08:01.435884+0000","last_became_active":"2023-11-24T06:08:01.433473+0000","last_became_peered":"2023-11-24T06:08:01.433473+0000","last_unstale":"2023-11-24T06:08:01.435884+0000","last_undegraded":"2023-11-24T06:08:01.435884+0000","last_fullsized":"2023-11-24T06:08:01.435884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:29.649298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433066+0000","last_change":"2023-11-24T06:08:01.428520+0000","last_active":"2023-11-24T06:08:01.433066+0000","last_peered":"2023-11-24T06:08:01.433066+0000","last_clean":"2023-11-24T06:08:01.433066+0000","last_became_active":"2023-11-24T06:08:01.428236+0000","last_became_peered":"2023-11-24T06:08:01.428236+0000","last_unstale":"2023-11-24T06:08:01.433066+0000","last_undegraded":"2023-11-24T06:08:01.433066+0000","last_fullsized":"2023-11-24T06:08:01.433066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:16.537475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436403+0000","last_change":"2023-11-24T06:08:01.433972+0000","last_active":"2023-11-24T06:08:01.436403+0000","last_peered":"2023-11-24T06:08:01.436403+0000","last_clean":"2023-11-24T06:08:01.436403+0000","last_became_active":"2023-11-24T06:08:01.433685+0000","last_became_peered":"2023-11-24T06:08:01.433685+0000","last_unstale":"2023-11-24T06:08:01.436403+0000","last_undegraded":"2023-11-24T06:08:01.436403+0000","last_fullsized":"2023-11-24T06:08:01.436403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:16.537475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437386+0000","last_change":"2023-11-24T06:08:01.435802+0000","last_active":"2023-11-24T06:08:01.437386+0000","last_peered":"2023-11-24T06:08:01.437386+0000","last_clean":"2023-11-24T06:08:01.437386+0000","last_became_active":"2023-11-24T06:08:01.435228+0000","last_became_peered":"2023-11-24T06:08:01.435228+0000","last_unstale":"2023-11-24T06:08:01.437386+0000","last_undegraded":"2023-11-24T06:08:01.437386+0000","last_fullsized":"2023-11-24T06:08:01.437386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:37:39.086654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433575+0000","last_change":"2023-11-24T06:08:01.432295+0000","last_active":"2023-11-24T06:08:01.433575+0000","last_peered":"2023-11-24T06:08:01.433575+0000","last_clean":"2023-11-24T06:08:01.433575+0000","last_became_active":"2023-11-24T06:08:01.431767+0000","last_became_peered":"2023-11-24T06:08:01.431767+0000","last_unstale":"2023-11-24T06:08:01.433575+0000","last_undegraded":"2023-11-24T06:08:01.433575+0000","last_fullsized":"2023-11-24T06:08:01.433575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:18.148689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437425+0000","last_change":"2023-11-24T06:08:01.434132+0000","last_active":"2023-11-24T06:08:01.437425+0000","last_peered":"2023-11-24T06:08:01.437425+0000","last_clean":"2023-11-24T06:08:01.437425+0000","last_became_active":"2023-11-24T06:08:01.433678+0000","last_became_peered":"2023-11-24T06:08:01.433678+0000","last_unstale":"2023-11-24T06:08:01.437425+0000","last_undegraded":"2023-11-24T06:08:01.437425+0000","last_fullsized":"2023-11-24T06:08:01.437425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:32.624231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.432819+0000","last_change":"2023-11-24T06:08:01.424409+0000","last_active":"2023-11-24T06:08:01.432819+0000","last_peered":"2023-11-24T06:08:01.432819+0000","last_clean":"2023-11-24T06:08:01.432819+0000","last_became_active":"2023-11-24T06:08:01.424152+0000","last_became_peered":"2023-11-24T06:08:01.424152+0000","last_unstale":"2023-11-24T06:08:01.432819+0000","last_undegraded":"2023-11-24T06:08:01.432819+0000","last_fullsized":"2023-11-24T06:08:01.432819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:32.624231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.438718+0000","last_change":"2023-11-24T06:08:01.436895+0000","last_active":"2023-11-24T06:08:01.438718+0000","last_peered":"2023-11-24T06:08:01.438718+0000","last_clean":"2023-11-24T06:08:01.438718+0000","last_became_active":"2023-11-24T06:08:01.436631+0000","last_became_peered":"2023-11-24T06:08:01.436631+0000","last_unstale":"2023-11-24T06:08:01.438718+0000","last_undegraded":"2023-11-24T06:08:01.438718+0000","last_fullsized":"2023-11-24T06:08:01.438718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:32.299407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436300+0000","last_change":"2023-11-24T06:08:01.435442+0000","last_active":"2023-11-24T06:08:01.436300+0000","last_peered":"2023-11-24T06:08:01.436300+0000","last_clean":"2023-11-24T06:08:01.436300+0000","last_became_active":"2023-11-24T06:08:01.435135+0000","last_became_peered":"2023-11-24T06:08:01.435135+0000","last_unstale":"2023-11-24T06:08:01.436300+0000","last_undegraded":"2023-11-24T06:08:01.436300+0000","last_fullsized":"2023-11-24T06:08:01.436300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:37:39.086654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433058+0000","last_change":"2023-11-24T06:08:01.423313+0000","last_active":"2023-11-24T06:08:01.433058+0000","last_peered":"2023-11-24T06:08:01.433058+0000","last_clean":"2023-11-24T06:08:01.433058+0000","last_became_active":"2023-11-24T06:08:01.423042+0000","last_became_peered":"2023-11-24T06:08:01.423042+0000","last_unstale":"2023-11-24T06:08:01.433058+0000","last_undegraded":"2023-11-24T06:08:01.433058+0000","last_fullsized":"2023-11-24T06:08:01.433058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:32.299407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":655560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":57835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":64770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":798720,"data_stored":655616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":66222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":64700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:08:05.833 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:08:05.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.984+0000 7f047c2f3700 1 Processor -- start 2023-11-24T06:08:05.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.984+0000 7f047c2f3700 1 -- start start 2023-11-24T06:08:05.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.985+0000 7f047c2f3700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f047406fc50 0x7f047406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.985+0000 7f047c2f3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f04740b0e90 0x7f047406ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.985+0000 7f047c2f3700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04740d1230 con 0x7f047415e1c0 2023-11-24T06:08:05.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.985+0000 7f047c2f3700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04740d0c70 con 0x7f047406fc50 2023-11-24T06:08:05.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.985+0000 7f047c2f3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04740d0f50 con 0x7f04740b0e90 2023-11-24T06:08:05.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.985+0000 7f047a08f700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f047415e1c0 0x7f047406f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52696/0 (socket says 172.21.15.91:52696) 2023-11-24T06:08:05.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.985+0000 7f047a08f700 1 -- 172.21.15.91:0/293798979 learned_addr learned my addr 172.21.15.91:0/293798979 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:05.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047908d700 1 -- 172.21.15.91:0/293798979 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581529857 0 0) 0x7f04740d1230 con 0x7f047415e1c0 2023-11-24T06:08:05.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047908d700 1 -- 172.21.15.91:0/293798979 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04600035b0 con 0x7f047415e1c0 2023-11-24T06:08:05.993 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047988e700 1 --2- 172.21.15.91:0/293798979 >> v2:172.21.15.94:3300/0 conn(0x7f047406fc50 0x7f047406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.993 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047a890700 1 --2- 172.21.15.91:0/293798979 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f04740b0e90 0x7f047406ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.993 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047908d700 1 -- 172.21.15.91:0/293798979 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3846823049 0 0) 0x7f04600035b0 con 0x7f047415e1c0 2023-11-24T06:08:05.993 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047908d700 1 -- 172.21.15.91:0/293798979 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f04740b0e90 msgr2=0x7f047406ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:05.993 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047908d700 1 --2- 172.21.15.91:0/293798979 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f04740b0e90 0x7f047406ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.993 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047908d700 1 -- 172.21.15.91:0/293798979 >> v2:172.21.15.94:3300/0 conn(0x7f047406fc50 msgr2=0x7f047406eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:05.993 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047908d700 1 --2- 172.21.15.91:0/293798979 >> v2:172.21.15.94:3300/0 conn(0x7f047406fc50 0x7f047406eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.993 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047908d700 1 -- 172.21.15.91:0/293798979 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f047406c210 con 0x7f047415e1c0 2023-11-24T06:08:05.993 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047908d700 1 -- 172.21.15.91:0/293798979 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f0464001670 con 0x7f047415e1c0 2023-11-24T06:08:05.993 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047908d700 1 -- 172.21.15.91:0/293798979 >> v1:172.21.15.91:6789/0 conn(0x7f047415e1c0 legacy=0x7f047406f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.993 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047908d700 1 --2- 172.21.15.91:0/293798979 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0460003a00 0x7f0460003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047908d700 1 --2- 172.21.15.91:0/293798979 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f04740b0e90 0x7f0460004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047908d700 1 --2- 172.21.15.91:0/293798979 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0460004d90 0x7f0460005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047908d700 1 -- 172.21.15.91:0/293798979 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04740d0c70 con 0x7f0460004d90 2023-11-24T06:08:05.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047908d700 1 -- 172.21.15.91:0/293798979 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04740d0f50 con 0x7f0460003a00 2023-11-24T06:08:05.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047908d700 1 -- 172.21.15.91:0/293798979 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046000ba10 con 0x7f04740b0e90 2023-11-24T06:08:05.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047a890700 1 --2- 172.21.15.91:0/293798979 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0460004d90 0x7f0460005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047a08f700 1 --2- 172.21.15.91:0/293798979 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0460003a00 0x7f0460003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047988e700 1 --2- 172.21.15.91:0/293798979 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f04740b0e90 0x7f0460004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:05.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047a890700 1 -- 172.21.15.91:0/293798979 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f04740b0e90 msgr2=0x7f0460004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:05.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047a890700 1 --2- 172.21.15.91:0/293798979 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f04740b0e90 0x7f0460004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047a890700 1 -- 172.21.15.91:0/293798979 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0460003a00 msgr2=0x7f0460003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:05.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047a890700 1 --2- 172.21.15.91:0/293798979 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0460003a00 0x7f0460003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:05.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047a890700 1 -- 172.21.15.91:0/293798979 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04600035b0 con 0x7f0460004d90 2023-11-24T06:08:05.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.986+0000 7f047a890700 1 --2- 172.21.15.91:0/293798979 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0460004d90 0x7f0460005210 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f046c009f90 tx=0x7f0460005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea49830757507920 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:05.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.987+0000 7f047908d700 1 -- 172.21.15.91:0/293798979 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c00ec30 con 0x7f0460004d90 2023-11-24T06:08:05.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.987+0000 7f047908d700 1 -- 172.21.15.91:0/293798979 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046c00f040 con 0x7f0460004d90 2023-11-24T06:08:05.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.987+0000 7f047908d700 1 -- 172.21.15.91:0/293798979 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c014480 con 0x7f0460004d90 2023-11-24T06:08:05.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.987+0000 7f047c2f3700 1 -- 172.21.15.91:0/293798979 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0460004d90 msgr2=0x7f0460005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:05.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.987+0000 7f047c2f3700 1 --2- 172.21.15.91:0/293798979 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0460004d90 0x7f0460005210 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f046c009f90 tx=0x7f0460005880 comp rx=0 tx=0).stop 2023-11-24T06:08:05.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.987+0000 7f047a890700 1 -- 172.21.15.91:0/293798979 reap_dead start 2023-11-24T06:08:05.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.987+0000 7f047c2f3700 1 -- 172.21.15.91:0/293798979 shutdown_connections 2023-11-24T06:08:05.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.987+0000 7f047c2f3700 1 -- 172.21.15.91:0/293798979 >> 172.21.15.91:0/293798979 conn(0x7f04740b3720 msgr2=0x7f04740b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:05.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.987+0000 7f047c2f3700 1 -- 172.21.15.91:0/293798979 shutdown_connections 2023-11-24T06:08:05.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.987+0000 7f047c2f3700 1 -- 172.21.15.91:0/293798979 wait complete. 2023-11-24T06:08:05.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.989+0000 7f047c2f3700 1 Processor -- start 2023-11-24T06:08:05.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.989+0000 7f047c2f3700 1 -- start start 2023-11-24T06:08:05.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047c2f3700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0460004d90 0x7f04740b6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047c2f3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f047414dcf0 0x7f04741f3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047c2f3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f047409d740 0x7f047409dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:05.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047c2f3700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047409e110 con 0x7f0460004d90 2023-11-24T06:08:05.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047c2f3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04740b73e0 con 0x7f047409d740 2023-11-24T06:08:05.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047c2f3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f047414e5b0 con 0x7f047414dcf0 2023-11-24T06:08:05.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047a08f700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0460004d90 0x7f04740b6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:06.000 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047a08f700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0460004d90 0x7f04740b6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57774/0 (socket says 172.21.15.91:57774) 2023-11-24T06:08:06.000 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047a08f700 1 -- 172.21.15.91:0/3439067900 learned_addr learned my addr 172.21.15.91:0/3439067900 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:06.000 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047988e700 1 --2- 172.21.15.91:0/3439067900 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f047414dcf0 0x7f04741f3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:06.000 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047a890700 1 --2- 172.21.15.91:0/3439067900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f047409d740 0x7f047409dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:06.000 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047a08f700 1 -- 172.21.15.91:0/3439067900 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f047414dcf0 msgr2=0x7f04741f3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:06.000 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047a08f700 1 --2- 172.21.15.91:0/3439067900 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f047414dcf0 0x7f04741f3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.000 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047a08f700 1 -- 172.21.15.91:0/3439067900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f047409d740 msgr2=0x7f047409dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:06.000 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047a08f700 1 --2- 172.21.15.91:0/3439067900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f047409d740 0x7f047409dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.000 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047a08f700 1 -- 172.21.15.91:0/3439067900 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f047414db00 con 0x7f0460004d90 2023-11-24T06:08:06.000 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047a890700 1 --2- 172.21.15.91:0/3439067900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f047409d740 0x7f047409dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:06.000 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047988e700 1 --2- 172.21.15.91:0/3439067900 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f047414dcf0 0x7f04741f3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:06.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047a08f700 1 --2- 172.21.15.91:0/3439067900 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0460004d90 0x7f04740b6ce0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0464000c00 tx=0x7f04640076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:06.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f046affd700 1 -- 172.21.15.91:0/3439067900 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046400f840 con 0x7f0460004d90 2023-11-24T06:08:06.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f046affd700 1 -- 172.21.15.91:0/3439067900 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046400f9c0 con 0x7f0460004d90 2023-11-24T06:08:06.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047c2f3700 1 -- 172.21.15.91:0/3439067900 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04741f3a40 con 0x7f0460004d90 2023-11-24T06:08:06.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f046affd700 1 -- 172.21.15.91:0/3439067900 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046400fd70 con 0x7f0460004d90 2023-11-24T06:08:06.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.990+0000 7f047c2f3700 1 -- 172.21.15.91:0/3439067900 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047409c8d0 con 0x7f0460004d90 2023-11-24T06:08:06.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.991+0000 7f047c2f3700 1 -- 172.21.15.91:0/3439067900 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0474068050 con 0x7f0460004d90 2023-11-24T06:08:06.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.992+0000 7f046affd700 1 -- 172.21.15.91:0/3439067900 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f046401d460 con 0x7f0460004d90 2023-11-24T06:08:06.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.992+0000 7f046affd700 1 --2- 172.21.15.91:0/3439067900 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f045806aed0 0x7f045806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:06.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.992+0000 7f046affd700 1 -- 172.21.15.91:0/3439067900 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04640895c0 con 0x7f0460004d90 2023-11-24T06:08:06.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.992+0000 7f047988e700 1 --2- 172.21.15.91:0/3439067900 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f045806aed0 0x7f045806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:06.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.993+0000 7f047988e700 1 --2- 172.21.15.91:0/3439067900 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f045806aed0 0x7f045806d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0470009ce0 tx=0x7f0470008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:06.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:05.997+0000 7f046affd700 1 -- 172.21.15.91:0/3439067900 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0464054700 con 0x7f0460004d90 2023-11-24T06:08:06.187 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.183+0000 7f047c2f3700 1 -- 172.21.15.91:0/3439067900 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f047415e5a0 con 0x7f045806aed0 2023-11-24T06:08:06.194 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.190+0000 7f046affd700 1 -- 172.21.15.91:0/3439067900 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264604 (secure 0 0 0) 0x7f0470009930 con 0x7f045806aed0 2023-11-24T06:08:06.197 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:06.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.193+0000 7f047c2f3700 1 -- 172.21.15.91:0/3439067900 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f045806aed0 msgr2=0x7f045806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:06.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.193+0000 7f047c2f3700 1 --2- 172.21.15.91:0/3439067900 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f045806aed0 0x7f045806d390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0470009ce0 tx=0x7f0470008040 comp rx=0 tx=0).stop 2023-11-24T06:08:06.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.193+0000 7f047c2f3700 1 -- 172.21.15.91:0/3439067900 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0460004d90 msgr2=0x7f04740b6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:06.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.193+0000 7f047c2f3700 1 --2- 172.21.15.91:0/3439067900 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0460004d90 0x7f04740b6ce0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0464000c00 tx=0x7f04640076a0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.193+0000 7f047c2f3700 1 -- 172.21.15.91:0/3439067900 shutdown_connections 2023-11-24T06:08:06.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.193+0000 7f047c2f3700 1 --2- 172.21.15.91:0/3439067900 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f045806aed0 0x7f045806d390 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.193+0000 7f047c2f3700 1 --2- 172.21.15.91:0/3439067900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f047409d740 0x7f047409dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.193+0000 7f047c2f3700 1 --2- 172.21.15.91:0/3439067900 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0460004d90 0x7f04740b6ce0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.193+0000 7f047c2f3700 1 --2- 172.21.15.91:0/3439067900 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f047414dcf0 0x7f04741f3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.201 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.193+0000 7f047c2f3700 1 -- 172.21.15.91:0/3439067900 >> 172.21.15.91:0/3439067900 conn(0x7f04740b3720 msgr2=0x7f04740b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:06.201 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.194+0000 7f047c2f3700 1 -- 172.21.15.91:0/3439067900 shutdown_connections 2023-11-24T06:08:06.201 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.194+0000 7f047c2f3700 1 -- 172.21.15.91:0/3439067900 wait complete. 2023-11-24T06:08:06.201 INFO:teuthology.orchestra.run.smithi091.stderr:dumped all 2023-11-24T06:08:06.209 INFO:teuthology.orchestra.run.smithi091.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-11-24T06:08:05.394031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":97,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218860,"kb_used_data":2692,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755860,"statfs":{"total":773094113280,"available":772870000640,"internally_reserved":0,"allocated":2756608,"data_stored":1693923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.993007"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436794+0000","last_change":"2023-11-24T06:08:01.435066+0000","last_active":"2023-11-24T06:08:01.436794+0000","last_peered":"2023-11-24T06:08:01.436794+0000","last_clean":"2023-11-24T06:08:01.436794+0000","last_became_active":"2023-11-24T06:08:01.434733+0000","last_became_peered":"2023-11-24T06:08:01.434733+0000","last_unstale":"2023-11-24T06:08:01.436794+0000","last_undegraded":"2023-11-24T06:08:01.436794+0000","last_fullsized":"2023-11-24T06:08:01.436794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:22.215190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.430750+0000","last_change":"2023-11-24T06:08:01.428588+0000","last_active":"2023-11-24T06:08:01.430750+0000","last_peered":"2023-11-24T06:08:01.430750+0000","last_clean":"2023-11-24T06:08:01.430750+0000","last_became_active":"2023-11-24T06:08:01.428257+0000","last_became_peered":"2023-11-24T06:08:01.428257+0000","last_unstale":"2023-11-24T06:08:01.430750+0000","last_undegraded":"2023-11-24T06:08:01.430750+0000","last_fullsized":"2023-11-24T06:08:01.430750+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:12.434134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.430257+0000","last_change":"2023-11-24T06:08:01.428960+0000","last_active":"2023-11-24T06:08:01.430257+0000","last_peered":"2023-11-24T06:08:01.430257+0000","last_clean":"2023-11-24T06:08:01.430257+0000","last_became_active":"2023-11-24T06:08:01.428600+0000","last_became_peered":"2023-11-24T06:08:01.428600+0000","last_unstale":"2023-11-24T06:08:01.430257+0000","last_undegraded":"2023-11-24T06:08:01.430257+0000","last_fullsized":"2023-11-24T06:08:01.430257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:32.299407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437709+0000","last_change":"2023-11-24T06:08:01.435529+0000","last_active":"2023-11-24T06:08:01.437709+0000","last_peered":"2023-11-24T06:08:01.437709+0000","last_clean":"2023-11-24T06:08:01.437709+0000","last_became_active":"2023-11-24T06:08:01.435128+0000","last_became_peered":"2023-11-24T06:08:01.435128+0000","last_unstale":"2023-11-24T06:08:01.437709+0000","last_undegraded":"2023-11-24T06:08:01.437709+0000","last_fullsized":"2023-11-24T06:08:01.437709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:18.148689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.435401+0000","last_change":"2023-11-24T06:08:01.434208+0000","last_active":"2023-11-24T06:08:01.435401+0000","last_peered":"2023-11-24T06:08:01.435401+0000","last_clean":"2023-11-24T06:08:01.435401+0000","last_became_active":"2023-11-24T06:08:01.433849+0000","last_became_peered":"2023-11-24T06:08:01.433849+0000","last_unstale":"2023-11-24T06:08:01.435401+0000","last_undegraded":"2023-11-24T06:08:01.435401+0000","last_fullsized":"2023-11-24T06:08:01.435401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:37:39.086654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437935+0000","last_change":"2023-11-24T06:08:01.436361+0000","last_active":"2023-11-24T06:08:01.437935+0000","last_peered":"2023-11-24T06:08:01.437935+0000","last_clean":"2023-11-24T06:08:01.437935+0000","last_became_active":"2023-11-24T06:08:01.434166+0000","last_became_peered":"2023-11-24T06:08:01.434166+0000","last_unstale":"2023-11-24T06:08:01.437935+0000","last_undegraded":"2023-11-24T06:08:01.437935+0000","last_fullsized":"2023-11-24T06:08:01.437935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:29.649298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.435270+0000","last_change":"2023-11-24T06:08:01.433541+0000","last_active":"2023-11-24T06:08:01.435270+0000","last_peered":"2023-11-24T06:08:01.435270+0000","last_clean":"2023-11-24T06:08:01.435270+0000","last_became_active":"2023-11-24T06:08:01.433237+0000","last_became_peered":"2023-11-24T06:08:01.433237+0000","last_unstale":"2023-11-24T06:08:01.435270+0000","last_undegraded":"2023-11-24T06:08:01.435270+0000","last_fullsized":"2023-11-24T06:08:01.435270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:21.084798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436565+0000","last_change":"2023-11-24T06:08:01.435678+0000","last_active":"2023-11-24T06:08:01.436565+0000","last_peered":"2023-11-24T06:08:01.436565+0000","last_clean":"2023-11-24T06:08:01.436565+0000","last_became_active":"2023-11-24T06:08:01.435365+0000","last_became_peered":"2023-11-24T06:08:01.435365+0000","last_unstale":"2023-11-24T06:08:01.436565+0000","last_undegraded":"2023-11-24T06:08:01.436565+0000","last_fullsized":"2023-11-24T06:08:01.436565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:29.649298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.439269+0000","last_change":"2023-11-24T06:08:01.438757+0000","last_active":"2023-11-24T06:08:01.439269+0000","last_peered":"2023-11-24T06:08:01.439269+0000","last_clean":"2023-11-24T06:08:01.439269+0000","last_became_active":"2023-11-24T06:08:01.438495+0000","last_became_peered":"2023-11-24T06:08:01.438495+0000","last_unstale":"2023-11-24T06:08:01.439269+0000","last_undegraded":"2023-11-24T06:08:01.439269+0000","last_fullsized":"2023-11-24T06:08:01.439269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:59.812992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437405+0000","last_change":"2023-11-24T06:08:01.435879+0000","last_active":"2023-11-24T06:08:01.437405+0000","last_peered":"2023-11-24T06:08:01.437405+0000","last_clean":"2023-11-24T06:08:01.437405+0000","last_became_active":"2023-11-24T06:08:01.435556+0000","last_became_peered":"2023-11-24T06:08:01.435556+0000","last_unstale":"2023-11-24T06:08:01.437405+0000","last_undegraded":"2023-11-24T06:08:01.437405+0000","last_fullsized":"2023-11-24T06:08:01.437405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:29.649298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437803+0000","last_change":"2023-11-24T06:08:01.436002+0000","last_active":"2023-11-24T06:08:01.437803+0000","last_peered":"2023-11-24T06:08:01.437803+0000","last_clean":"2023-11-24T06:08:01.437803+0000","last_became_active":"2023-11-24T06:08:01.435547+0000","last_became_peered":"2023-11-24T06:08:01.435547+0000","last_unstale":"2023-11-24T06:08:01.437803+0000","last_undegraded":"2023-11-24T06:08:01.437803+0000","last_fullsized":"2023-11-24T06:08:01.437803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:18.148689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.438585+0000","last_change":"2023-11-24T06:08:01.431781+0000","last_active":"2023-11-24T06:08:01.438585+0000","last_peered":"2023-11-24T06:08:01.438585+0000","last_clean":"2023-11-24T06:08:01.438585+0000","last_became_active":"2023-11-24T06:08:01.431494+0000","last_became_peered":"2023-11-24T06:08:01.431494+0000","last_unstale":"2023-11-24T06:08:01.438585+0000","last_undegraded":"2023-11-24T06:08:01.438585+0000","last_fullsized":"2023-11-24T06:08:01.438585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:29.048558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436150+0000","last_change":"2023-11-24T06:08:01.434455+0000","last_active":"2023-11-24T06:08:01.436150+0000","last_peered":"2023-11-24T06:08:01.436150+0000","last_clean":"2023-11-24T06:08:01.436150+0000","last_became_active":"2023-11-24T06:08:01.434201+0000","last_became_peered":"2023-11-24T06:08:01.434201+0000","last_unstale":"2023-11-24T06:08:01.436150+0000","last_undegraded":"2023-11-24T06:08:01.436150+0000","last_fullsized":"2023-11-24T06:08:01.436150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:40.365444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.430133+0000","last_change":"2023-11-24T06:08:01.427810+0000","last_active":"2023-11-24T06:08:01.430133+0000","last_peered":"2023-11-24T06:08:01.430133+0000","last_clean":"2023-11-24T06:08:01.430133+0000","last_became_active":"2023-11-24T06:08:01.427545+0000","last_became_peered":"2023-11-24T06:08:01.427545+0000","last_unstale":"2023-11-24T06:08:01.430133+0000","last_undegraded":"2023-11-24T06:08:01.430133+0000","last_fullsized":"2023-11-24T06:08:01.430133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:59.736889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.432709+0000","last_change":"2023-11-24T06:08:01.431162+0000","last_active":"2023-11-24T06:08:01.432709+0000","last_peered":"2023-11-24T06:08:01.432709+0000","last_clean":"2023-11-24T06:08:01.432709+0000","last_became_active":"2023-11-24T06:08:01.430862+0000","last_became_peered":"2023-11-24T06:08:01.430862+0000","last_unstale":"2023-11-24T06:08:01.432709+0000","last_undegraded":"2023-11-24T06:08:01.432709+0000","last_fullsized":"2023-11-24T06:08:01.432709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:32.299407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437407+0000","last_change":"2023-11-24T06:08:01.434078+0000","last_active":"2023-11-24T06:08:01.437407+0000","last_peered":"2023-11-24T06:08:01.437407+0000","last_clean":"2023-11-24T06:08:01.437407+0000","last_became_active":"2023-11-24T06:08:01.433636+0000","last_became_peered":"2023-11-24T06:08:01.433636+0000","last_unstale":"2023-11-24T06:08:01.437407+0000","last_undegraded":"2023-11-24T06:08:01.437407+0000","last_fullsized":"2023-11-24T06:08:01.437407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:42.556322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.438080+0000","last_change":"2023-11-24T06:08:01.436588+0000","last_active":"2023-11-24T06:08:01.438080+0000","last_peered":"2023-11-24T06:08:01.438080+0000","last_clean":"2023-11-24T06:08:01.438080+0000","last_became_active":"2023-11-24T06:08:01.436193+0000","last_became_peered":"2023-11-24T06:08:01.436193+0000","last_unstale":"2023-11-24T06:08:01.438080+0000","last_undegraded":"2023-11-24T06:08:01.438080+0000","last_fullsized":"2023-11-24T06:08:01.438080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:04.385452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425439+0000","last_change":"2023-11-24T06:08:01.424485+0000","last_active":"2023-11-24T06:08:01.425439+0000","last_peered":"2023-11-24T06:08:01.425439+0000","last_clean":"2023-11-24T06:08:01.425439+0000","last_became_active":"2023-11-24T06:08:01.424234+0000","last_became_peered":"2023-11-24T06:08:01.424234+0000","last_unstale":"2023-11-24T06:08:01.425439+0000","last_undegraded":"2023-11-24T06:08:01.425439+0000","last_fullsized":"2023-11-24T06:08:01.425439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:32.299407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.435713+0000","last_change":"2023-11-24T06:08:01.434319+0000","last_active":"2023-11-24T06:08:01.435713+0000","last_peered":"2023-11-24T06:08:01.435713+0000","last_clean":"2023-11-24T06:08:01.435713+0000","last_became_active":"2023-11-24T06:08:01.433984+0000","last_became_peered":"2023-11-24T06:08:01.433984+0000","last_unstale":"2023-11-24T06:08:01.435713+0000","last_undegraded":"2023-11-24T06:08:01.435713+0000","last_fullsized":"2023-11-24T06:08:01.435713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:52:57.033291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.738808+0000","last_change":"2023-11-24T06:08:00.418990+0000","last_active":"2023-11-24T06:08:01.738808+0000","last_peered":"2023-11-24T06:08:01.738808+0000","last_clean":"2023-11-24T06:08:01.738808+0000","last_became_active":"2023-11-24T06:08:00.418683+0000","last_became_peered":"2023-11-24T06:08:00.418683+0000","last_unstale":"2023-11-24T06:08:01.738808+0000","last_undegraded":"2023-11-24T06:08:01.738808+0000","last_fullsized":"2023-11-24T06:08:01.738808+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:13.152063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433540+0000","last_change":"2023-11-24T06:08:01.431913+0000","last_active":"2023-11-24T06:08:01.433540+0000","last_peered":"2023-11-24T06:08:01.433540+0000","last_clean":"2023-11-24T06:08:01.433540+0000","last_became_active":"2023-11-24T06:08:01.431373+0000","last_became_peered":"2023-11-24T06:08:01.431373+0000","last_unstale":"2023-11-24T06:08:01.433540+0000","last_undegraded":"2023-11-24T06:08:01.433540+0000","last_fullsized":"2023-11-24T06:08:01.433540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:12.434134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.738974+0000","last_change":"2023-11-24T06:08:00.424674+0000","last_active":"2023-11-24T06:08:01.738974+0000","last_peered":"2023-11-24T06:08:01.738974+0000","last_clean":"2023-11-24T06:08:01.738974+0000","last_became_active":"2023-11-24T06:08:00.424282+0000","last_became_peered":"2023-11-24T06:08:00.424282+0000","last_unstale":"2023-11-24T06:08:01.738974+0000","last_undegraded":"2023-11-24T06:08:01.738974+0000","last_fullsized":"2023-11-24T06:08:01.738974+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:03.380514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.435944+0000","last_change":"2023-11-24T06:08:01.434659+0000","last_active":"2023-11-24T06:08:01.435944+0000","last_peered":"2023-11-24T06:08:01.435944+0000","last_clean":"2023-11-24T06:08:01.435944+0000","last_became_active":"2023-11-24T06:08:01.434373+0000","last_became_peered":"2023-11-24T06:08:01.434373+0000","last_unstale":"2023-11-24T06:08:01.435944+0000","last_undegraded":"2023-11-24T06:08:01.435944+0000","last_fullsized":"2023-11-24T06:08:01.435944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:19.821244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.736658+0000","last_change":"2023-11-24T06:08:00.426586+0000","last_active":"2023-11-24T06:08:01.736658+0000","last_peered":"2023-11-24T06:08:01.736658+0000","last_clean":"2023-11-24T06:08:01.736658+0000","last_became_active":"2023-11-24T06:08:00.426279+0000","last_became_peered":"2023-11-24T06:08:00.426279+0000","last_unstale":"2023-11-24T06:08:01.736658+0000","last_undegraded":"2023-11-24T06:08:01.736658+0000","last_fullsized":"2023-11-24T06:08:01.736658+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:06:57.668726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.441384+0000","last_change":"2023-11-24T06:08:01.439887+0000","last_active":"2023-11-24T06:08:01.441384+0000","last_peered":"2023-11-24T06:08:01.441384+0000","last_clean":"2023-11-24T06:08:01.441384+0000","last_became_active":"2023-11-24T06:08:01.439606+0000","last_became_peered":"2023-11-24T06:08:01.439606+0000","last_unstale":"2023-11-24T06:08:01.441384+0000","last_undegraded":"2023-11-24T06:08:01.441384+0000","last_fullsized":"2023-11-24T06:08:01.441384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:52:57.033291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.737400+0000","last_change":"2023-11-24T06:08:00.423364+0000","last_active":"2023-11-24T06:08:01.737400+0000","last_peered":"2023-11-24T06:08:01.737400+0000","last_clean":"2023-11-24T06:08:01.737400+0000","last_became_active":"2023-11-24T06:08:00.423062+0000","last_became_peered":"2023-11-24T06:08:00.423062+0000","last_unstale":"2023-11-24T06:08:01.737400+0000","last_undegraded":"2023-11-24T06:08:01.737400+0000","last_fullsized":"2023-11-24T06:08:01.737400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:28.644360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.439749+0000","last_change":"2023-11-24T06:08:01.438369+0000","last_active":"2023-11-24T06:08:01.439749+0000","last_peered":"2023-11-24T06:08:01.439749+0000","last_clean":"2023-11-24T06:08:01.439749+0000","last_became_active":"2023-11-24T06:08:01.438112+0000","last_became_peered":"2023-11-24T06:08:01.438112+0000","last_unstale":"2023-11-24T06:08:01.439749+0000","last_undegraded":"2023-11-24T06:08:01.439749+0000","last_fullsized":"2023-11-24T06:08:01.439749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:40.365444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.738620+0000","last_change":"2023-11-24T06:08:00.425844+0000","last_active":"2023-11-24T06:08:01.738620+0000","last_peered":"2023-11-24T06:08:01.738620+0000","last_clean":"2023-11-24T06:08:01.738620+0000","last_became_active":"2023-11-24T06:08:00.425587+0000","last_became_peered":"2023-11-24T06:08:00.425587+0000","last_unstale":"2023-11-24T06:08:01.738620+0000","last_undegraded":"2023-11-24T06:08:01.738620+0000","last_fullsized":"2023-11-24T06:08:01.738620+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:29.195643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.434339+0000","last_change":"2023-11-24T06:08:01.427389+0000","last_active":"2023-11-24T06:08:01.434339+0000","last_peered":"2023-11-24T06:08:01.434339+0000","last_clean":"2023-11-24T06:08:01.434339+0000","last_became_active":"2023-11-24T06:08:01.427038+0000","last_became_peered":"2023-11-24T06:08:01.427038+0000","last_unstale":"2023-11-24T06:08:01.434339+0000","last_undegraded":"2023-11-24T06:08:01.434339+0000","last_fullsized":"2023-11-24T06:08:01.434339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:19.821244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.735862+0000","last_change":"2023-11-24T06:08:00.411792+0000","last_active":"2023-11-24T06:08:01.735862+0000","last_peered":"2023-11-24T06:08:01.735862+0000","last_clean":"2023-11-24T06:08:01.735862+0000","last_became_active":"2023-11-24T06:08:00.411486+0000","last_became_peered":"2023-11-24T06:08:00.411486+0000","last_unstale":"2023-11-24T06:08:01.735862+0000","last_undegraded":"2023-11-24T06:08:01.735862+0000","last_fullsized":"2023-11-24T06:08:01.735862+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:41.551384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.445646+0000","last_change":"2023-11-24T06:08:01.438231+0000","last_active":"2023-11-24T06:08:01.445646+0000","last_peered":"2023-11-24T06:08:01.445646+0000","last_clean":"2023-11-24T06:08:01.445646+0000","last_became_active":"2023-11-24T06:08:01.437944+0000","last_became_peered":"2023-11-24T06:08:01.437944+0000","last_unstale":"2023-11-24T06:08:01.445646+0000","last_undegraded":"2023-11-24T06:08:01.445646+0000","last_fullsized":"2023-11-24T06:08:01.445646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:16.537475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.421698+0000","last_change":"2023-11-24T06:08:00.416967+0000","last_active":"2023-11-24T06:08:01.421698+0000","last_peered":"2023-11-24T06:08:01.421698+0000","last_clean":"2023-11-24T06:08:01.421698+0000","last_became_active":"2023-11-24T06:08:00.416532+0000","last_became_peered":"2023-11-24T06:08:00.416532+0000","last_unstale":"2023-11-24T06:08:01.421698+0000","last_undegraded":"2023-11-24T06:08:01.421698+0000","last_fullsized":"2023-11-24T06:08:01.421698+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:20.079860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433288+0000","last_change":"2023-11-24T06:08:01.429802+0000","last_active":"2023-11-24T06:08:01.433288+0000","last_peered":"2023-11-24T06:08:01.433288+0000","last_clean":"2023-11-24T06:08:01.433288+0000","last_became_active":"2023-11-24T06:08:01.429578+0000","last_became_peered":"2023-11-24T06:08:01.429578+0000","last_unstale":"2023-11-24T06:08:01.433288+0000","last_undegraded":"2023-11-24T06:08:01.433288+0000","last_fullsized":"2023-11-24T06:08:01.433288+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:19.821244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.427448+0000","last_change":"2023-11-24T06:08:00.422030+0000","last_active":"2023-11-24T06:08:01.427448+0000","last_peered":"2023-11-24T06:08:01.427448+0000","last_clean":"2023-11-24T06:08:01.427448+0000","last_became_active":"2023-11-24T06:08:00.421740+0000","last_became_peered":"2023-11-24T06:08:00.421740+0000","last_unstale":"2023-11-24T06:08:01.427448+0000","last_undegraded":"2023-11-24T06:08:01.427448+0000","last_fullsized":"2023-11-24T06:08:01.427448+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:58.808054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436372+0000","last_change":"2023-11-24T06:08:01.433496+0000","last_active":"2023-11-24T06:08:01.436372+0000","last_peered":"2023-11-24T06:08:01.436372+0000","last_clean":"2023-11-24T06:08:01.436372+0000","last_became_active":"2023-11-24T06:08:01.433238+0000","last_became_peered":"2023-11-24T06:08:01.433238+0000","last_unstale":"2023-11-24T06:08:01.436372+0000","last_undegraded":"2023-11-24T06:08:01.436372+0000","last_fullsized":"2023-11-24T06:08:01.436372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:14.157001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425023+0000","last_change":"2023-11-24T06:08:00.422595+0000","last_active":"2023-11-24T06:08:01.425023+0000","last_peered":"2023-11-24T06:08:01.425023+0000","last_clean":"2023-11-24T06:08:01.425023+0000","last_became_active":"2023-11-24T06:08:00.422365+0000","last_became_peered":"2023-11-24T06:08:00.422365+0000","last_unstale":"2023-11-24T06:08:01.425023+0000","last_undegraded":"2023-11-24T06:08:01.425023+0000","last_fullsized":"2023-11-24T06:08:01.425023+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:39.360506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437390+0000","last_change":"2023-11-24T06:08:01.434890+0000","last_active":"2023-11-24T06:08:01.437390+0000","last_peered":"2023-11-24T06:08:01.437390+0000","last_clean":"2023-11-24T06:08:01.437390+0000","last_became_active":"2023-11-24T06:08:01.434706+0000","last_became_peered":"2023-11-24T06:08:01.434706+0000","last_unstale":"2023-11-24T06:08:01.437390+0000","last_undegraded":"2023-11-24T06:08:01.437390+0000","last_fullsized":"2023-11-24T06:08:01.437390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:21.084798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425886+0000","last_change":"2023-11-24T06:08:00.423297+0000","last_active":"2023-11-24T06:08:01.425886+0000","last_peered":"2023-11-24T06:08:01.425886+0000","last_clean":"2023-11-24T06:08:01.425886+0000","last_became_active":"2023-11-24T06:08:00.423028+0000","last_became_peered":"2023-11-24T06:08:00.423028+0000","last_unstale":"2023-11-24T06:08:01.425886+0000","last_undegraded":"2023-11-24T06:08:01.425886+0000","last_fullsized":"2023-11-24T06:08:01.425886+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:37:38.081716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.440137+0000","last_change":"2023-11-24T06:08:01.437725+0000","last_active":"2023-11-24T06:08:01.440137+0000","last_peered":"2023-11-24T06:08:01.440137+0000","last_clean":"2023-11-24T06:08:01.440137+0000","last_became_active":"2023-11-24T06:08:01.437464+0000","last_became_peered":"2023-11-24T06:08:01.437464+0000","last_unstale":"2023-11-24T06:08:01.440137+0000","last_undegraded":"2023-11-24T06:08:01.440137+0000","last_fullsized":"2023-11-24T06:08:01.440137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:21:46.421251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.737950+0000","last_change":"2023-11-24T06:08:00.421041+0000","last_active":"2023-11-24T06:08:01.737950+0000","last_peered":"2023-11-24T06:08:01.737950+0000","last_clean":"2023-11-24T06:08:01.737950+0000","last_became_active":"2023-11-24T06:08:00.420677+0000","last_became_peered":"2023-11-24T06:08:00.420677+0000","last_unstale":"2023-11-24T06:08:01.737950+0000","last_undegraded":"2023-11-24T06:08:01.737950+0000","last_fullsized":"2023-11-24T06:08:01.737950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:29.195643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.439962+0000","last_change":"2023-11-24T06:08:01.437971+0000","last_active":"2023-11-24T06:08:01.439962+0000","last_peered":"2023-11-24T06:08:01.439962+0000","last_clean":"2023-11-24T06:08:01.439962+0000","last_became_active":"2023-11-24T06:08:01.437640+0000","last_became_peered":"2023-11-24T06:08:01.437640+0000","last_unstale":"2023-11-24T06:08:01.439962+0000","last_undegraded":"2023-11-24T06:08:01.439962+0000","last_fullsized":"2023-11-24T06:08:01.439962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:37:59.736889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.741721+0000","last_change":"2023-11-24T06:08:00.418490+0000","last_active":"2023-11-24T06:08:01.741721+0000","last_peered":"2023-11-24T06:08:01.741721+0000","last_clean":"2023-11-24T06:08:01.741721+0000","last_became_active":"2023-11-24T06:08:00.418200+0000","last_became_peered":"2023-11-24T06:08:00.418200+0000","last_unstale":"2023-11-24T06:08:01.741721+0000","last_undegraded":"2023-11-24T06:08:01.741721+0000","last_fullsized":"2023-11-24T06:08:01.741721+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:03.380514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.442296+0000","last_change":"2023-11-24T06:08:01.439461+0000","last_active":"2023-11-24T06:08:01.442296+0000","last_peered":"2023-11-24T06:08:01.442296+0000","last_clean":"2023-11-24T06:08:01.442296+0000","last_became_active":"2023-11-24T06:08:01.439210+0000","last_became_peered":"2023-11-24T06:08:01.439210+0000","last_unstale":"2023-11-24T06:08:01.442296+0000","last_undegraded":"2023-11-24T06:08:01.442296+0000","last_fullsized":"2023-11-24T06:08:01.442296+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:28.564398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.736609+0000","last_change":"2023-11-24T06:08:00.423172+0000","last_active":"2023-11-24T06:08:01.736609+0000","last_peered":"2023-11-24T06:08:01.736609+0000","last_clean":"2023-11-24T06:08:01.736609+0000","last_became_active":"2023-11-24T06:08:00.422899+0000","last_became_peered":"2023-11-24T06:08:00.422899+0000","last_unstale":"2023-11-24T06:08:01.736609+0000","last_undegraded":"2023-11-24T06:08:01.736609+0000","last_fullsized":"2023-11-24T06:08:01.736609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:17.143751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436314+0000","last_change":"2023-11-24T06:08:01.435150+0000","last_active":"2023-11-24T06:08:01.436314+0000","last_peered":"2023-11-24T06:08:01.436314+0000","last_clean":"2023-11-24T06:08:01.436314+0000","last_became_active":"2023-11-24T06:08:01.434861+0000","last_became_peered":"2023-11-24T06:08:01.434861+0000","last_unstale":"2023-11-24T06:08:01.436314+0000","last_undegraded":"2023-11-24T06:08:01.436314+0000","last_fullsized":"2023-11-24T06:08:01.436314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:32.624231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.429337+0000","last_change":"2023-11-24T06:08:00.422660+0000","last_active":"2023-11-24T06:08:01.429337+0000","last_peered":"2023-11-24T06:08:01.429337+0000","last_clean":"2023-11-24T06:08:01.429337+0000","last_became_active":"2023-11-24T06:08:00.421922+0000","last_became_peered":"2023-11-24T06:08:00.421922+0000","last_unstale":"2023-11-24T06:08:01.429337+0000","last_undegraded":"2023-11-24T06:08:01.429337+0000","last_fullsized":"2023-11-24T06:08:01.429337+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:27.559460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.443240+0000","last_change":"2023-11-24T06:08:01.442581+0000","last_active":"2023-11-24T06:08:01.443240+0000","last_peered":"2023-11-24T06:08:01.443240+0000","last_clean":"2023-11-24T06:08:01.443240+0000","last_became_active":"2023-11-24T06:08:01.442145+0000","last_became_peered":"2023-11-24T06:08:01.442145+0000","last_unstale":"2023-11-24T06:08:01.443240+0000","last_undegraded":"2023-11-24T06:08:01.443240+0000","last_fullsized":"2023-11-24T06:08:01.443240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:04.385452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.419830+0000","last_change":"2023-11-24T06:08:00.413903+0000","last_active":"2023-11-24T06:08:01.419830+0000","last_peered":"2023-11-24T06:08:01.419830+0000","last_clean":"2023-11-24T06:08:01.419830+0000","last_became_active":"2023-11-24T06:08:00.413581+0000","last_became_peered":"2023-11-24T06:08:00.413581+0000","last_unstale":"2023-11-24T06:08:01.419830+0000","last_undegraded":"2023-11-24T06:08:01.419830+0000","last_fullsized":"2023-11-24T06:08:01.419830+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:15.532537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433217+0000","last_change":"2023-11-24T06:08:01.429649+0000","last_active":"2023-11-24T06:08:01.433217+0000","last_peered":"2023-11-24T06:08:01.433217+0000","last_clean":"2023-11-24T06:08:01.433217+0000","last_became_active":"2023-11-24T06:08:01.429189+0000","last_became_peered":"2023-11-24T06:08:01.429189+0000","last_unstale":"2023-11-24T06:08:01.433217+0000","last_undegraded":"2023-11-24T06:08:01.433217+0000","last_fullsized":"2023-11-24T06:08:01.433217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:21.084798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.739985+0000","last_change":"2023-11-24T06:08:00.417220+0000","last_active":"2023-11-24T06:08:01.739985+0000","last_peered":"2023-11-24T06:08:01.739985+0000","last_clean":"2023-11-24T06:08:01.739985+0000","last_became_active":"2023-11-24T06:08:00.416947+0000","last_became_peered":"2023-11-24T06:08:00.416947+0000","last_unstale":"2023-11-24T06:08:01.739985+0000","last_undegraded":"2023-11-24T06:08:01.739985+0000","last_fullsized":"2023-11-24T06:08:01.739985+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:18.816306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.442913+0000","last_change":"2023-11-24T06:08:01.442182+0000","last_active":"2023-11-24T06:08:01.442913+0000","last_peered":"2023-11-24T06:08:01.442913+0000","last_clean":"2023-11-24T06:08:01.442913+0000","last_became_active":"2023-11-24T06:08:01.441895+0000","last_became_peered":"2023-11-24T06:08:01.441895+0000","last_unstale":"2023-11-24T06:08:01.442913+0000","last_undegraded":"2023-11-24T06:08:01.442913+0000","last_fullsized":"2023-11-24T06:08:01.442913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:47:16.508162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.738716+0000","last_change":"2023-11-24T06:08:00.420431+0000","last_active":"2023-11-24T06:08:01.738716+0000","last_peered":"2023-11-24T06:08:01.738716+0000","last_clean":"2023-11-24T06:08:01.738716+0000","last_became_active":"2023-11-24T06:08:00.420108+0000","last_became_peered":"2023-11-24T06:08:00.420108+0000","last_unstale":"2023-11-24T06:08:01.738716+0000","last_undegraded":"2023-11-24T06:08:01.738716+0000","last_fullsized":"2023-11-24T06:08:01.738716+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:11.429196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.442887+0000","last_change":"2023-11-24T06:08:01.441968+0000","last_active":"2023-11-24T06:08:01.442887+0000","last_peered":"2023-11-24T06:08:01.442887+0000","last_clean":"2023-11-24T06:08:01.442887+0000","last_became_active":"2023-11-24T06:08:01.440920+0000","last_became_peered":"2023-11-24T06:08:01.440920+0000","last_unstale":"2023-11-24T06:08:01.442887+0000","last_undegraded":"2023-11-24T06:08:01.442887+0000","last_fullsized":"2023-11-24T06:08:01.442887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:40:20.503614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425271+0000","last_change":"2023-11-24T06:08:00.423440+0000","last_active":"2023-11-24T06:08:01.425271+0000","last_peered":"2023-11-24T06:08:01.425271+0000","last_clean":"2023-11-24T06:08:01.425271+0000","last_became_active":"2023-11-24T06:08:00.423175+0000","last_became_peered":"2023-11-24T06:08:00.423175+0000","last_unstale":"2023-11-24T06:08:01.425271+0000","last_undegraded":"2023-11-24T06:08:01.425271+0000","last_fullsized":"2023-11-24T06:08:01.425271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:39.360506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437509+0000","last_change":"2023-11-24T06:08:01.435280+0000","last_active":"2023-11-24T06:08:01.437509+0000","last_peered":"2023-11-24T06:08:01.437509+0000","last_clean":"2023-11-24T06:08:01.437509+0000","last_became_active":"2023-11-24T06:08:01.434963+0000","last_became_peered":"2023-11-24T06:08:01.434963+0000","last_unstale":"2023-11-24T06:08:01.437509+0000","last_undegraded":"2023-11-24T06:08:01.437509+0000","last_fullsized":"2023-11-24T06:08:01.437509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:16.537475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.428526+0000","last_change":"2023-11-24T06:08:00.420129+0000","last_active":"2023-11-24T06:08:01.428526+0000","last_peered":"2023-11-24T06:08:01.428526+0000","last_clean":"2023-11-24T06:08:01.428526+0000","last_became_active":"2023-11-24T06:08:00.418612+0000","last_became_peered":"2023-11-24T06:08:00.418612+0000","last_unstale":"2023-11-24T06:08:01.428526+0000","last_undegraded":"2023-11-24T06:08:01.428526+0000","last_fullsized":"2023-11-24T06:08:01.428526+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:40:19.498676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.439553+0000","last_change":"2023-11-24T06:08:01.430171+0000","last_active":"2023-11-24T06:08:01.439553+0000","last_peered":"2023-11-24T06:08:01.439553+0000","last_clean":"2023-11-24T06:08:01.439553+0000","last_became_active":"2023-11-24T06:08:01.428562+0000","last_became_peered":"2023-11-24T06:08:01.428562+0000","last_unstale":"2023-11-24T06:08:01.439553+0000","last_undegraded":"2023-11-24T06:08:01.439553+0000","last_fullsized":"2023-11-24T06:08:01.439553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:32.624231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436350+0000","last_change":"2023-11-24T06:08:01.433086+0000","last_active":"2023-11-24T06:08:01.436350+0000","last_peered":"2023-11-24T06:08:01.436350+0000","last_clean":"2023-11-24T06:08:01.436350+0000","last_became_active":"2023-11-24T06:08:01.432747+0000","last_became_peered":"2023-11-24T06:08:01.432747+0000","last_unstale":"2023-11-24T06:08:01.436350+0000","last_undegraded":"2023-11-24T06:08:01.436350+0000","last_fullsized":"2023-11-24T06:08:01.436350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:42.556322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.417666+0000","last_change":"2023-11-24T06:08:00.416138+0000","last_active":"2023-11-24T06:08:01.417666+0000","last_peered":"2023-11-24T06:08:01.417666+0000","last_clean":"2023-11-24T06:08:01.417666+0000","last_became_active":"2023-11-24T06:08:00.415909+0000","last_became_peered":"2023-11-24T06:08:00.415909+0000","last_unstale":"2023-11-24T06:08:01.417666+0000","last_undegraded":"2023-11-24T06:08:01.417666+0000","last_fullsized":"2023-11-24T06:08:01.417666+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:31.619293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.445658+0000","last_change":"2023-11-24T06:08:01.438377+0000","last_active":"2023-11-24T06:08:01.445658+0000","last_peered":"2023-11-24T06:08:01.445658+0000","last_clean":"2023-11-24T06:08:01.445658+0000","last_became_active":"2023-11-24T06:08:01.438142+0000","last_became_peered":"2023-11-24T06:08:01.438142+0000","last_unstale":"2023-11-24T06:08:01.445658+0000","last_undegraded":"2023-11-24T06:08:01.445658+0000","last_fullsized":"2023-11-24T06:08:01.445658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:37:39.086654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.423775+0000","last_change":"2023-11-24T06:08:00.419708+0000","last_active":"2023-11-24T06:08:01.423775+0000","last_peered":"2023-11-24T06:08:01.423775+0000","last_clean":"2023-11-24T06:08:01.423775+0000","last_became_active":"2023-11-24T06:08:00.419444+0000","last_became_peered":"2023-11-24T06:08:00.419444+0000","last_unstale":"2023-11-24T06:08:01.423775+0000","last_undegraded":"2023-11-24T06:08:01.423775+0000","last_fullsized":"2023-11-24T06:08:01.423775+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:13.152063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-11-24T06:08:00.417748+0000","last_change":"2023-11-24T06:07:49.463719+0000","last_active":"2023-11-24T06:08:00.417748+0000","last_peered":"2023-11-24T06:08:00.417748+0000","last_clean":"2023-11-24T06:08:00.417748+0000","last_became_active":"2023-11-24T06:07:49.462736+0000","last_became_peered":"2023-11-24T06:07:49.462736+0000","last_unstale":"2023-11-24T06:08:00.417748+0000","last_undegraded":"2023-11-24T06:08:00.417748+0000","last_fullsized":"2023-11-24T06:08:00.417748+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_clean_scrub_stamp":"2023-11-24T06:07:47.860604+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:16.508770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.445621+0000","last_change":"2023-11-24T06:08:01.438231+0000","last_active":"2023-11-24T06:08:01.445621+0000","last_peered":"2023-11-24T06:08:01.445621+0000","last_clean":"2023-11-24T06:08:01.445621+0000","last_became_active":"2023-11-24T06:08:01.437970+0000","last_became_peered":"2023-11-24T06:08:01.437970+0000","last_unstale":"2023-11-24T06:08:01.445621+0000","last_undegraded":"2023-11-24T06:08:01.445621+0000","last_fullsized":"2023-11-24T06:08:01.445621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:42.556322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425496+0000","last_change":"2023-11-24T06:08:00.423316+0000","last_active":"2023-11-24T06:08:01.425496+0000","last_peered":"2023-11-24T06:08:01.425496+0000","last_clean":"2023-11-24T06:08:01.425496+0000","last_became_active":"2023-11-24T06:08:00.423046+0000","last_became_peered":"2023-11-24T06:08:00.423046+0000","last_unstale":"2023-11-24T06:08:01.425496+0000","last_undegraded":"2023-11-24T06:08:01.425496+0000","last_fullsized":"2023-11-24T06:08:01.425496+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:17.143751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.429067+0000","last_change":"2023-11-24T06:08:00.424803+0000","last_active":"2023-11-24T06:08:01.429067+0000","last_peered":"2023-11-24T06:08:01.429067+0000","last_clean":"2023-11-24T06:08:01.429067+0000","last_became_active":"2023-11-24T06:08:00.424473+0000","last_became_peered":"2023-11-24T06:08:00.424473+0000","last_unstale":"2023-11-24T06:08:01.429067+0000","last_undegraded":"2023-11-24T06:08:01.429067+0000","last_fullsized":"2023-11-24T06:08:01.429067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:00.828006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.432957+0000","last_change":"2023-11-24T06:08:01.432093+0000","last_active":"2023-11-24T06:08:01.432957+0000","last_peered":"2023-11-24T06:08:01.432957+0000","last_clean":"2023-11-24T06:08:01.432957+0000","last_became_active":"2023-11-24T06:08:01.431795+0000","last_became_peered":"2023-11-24T06:08:01.431795+0000","last_unstale":"2023-11-24T06:08:01.432957+0000","last_undegraded":"2023-11-24T06:08:01.432957+0000","last_fullsized":"2023-11-24T06:08:01.432957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:37:39.086654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425234+0000","last_change":"2023-11-24T06:08:00.416062+0000","last_active":"2023-11-24T06:08:01.425234+0000","last_peered":"2023-11-24T06:08:01.425234+0000","last_clean":"2023-11-24T06:08:01.425234+0000","last_became_active":"2023-11-24T06:08:00.415783+0000","last_became_peered":"2023-11-24T06:08:00.415783+0000","last_unstale":"2023-11-24T06:08:01.425234+0000","last_undegraded":"2023-11-24T06:08:01.425234+0000","last_fullsized":"2023-11-24T06:08:01.425234+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:03.380514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.445642+0000","last_change":"2023-11-24T06:08:01.437911+0000","last_active":"2023-11-24T06:08:01.445642+0000","last_peered":"2023-11-24T06:08:01.445642+0000","last_clean":"2023-11-24T06:08:01.445642+0000","last_became_active":"2023-11-24T06:08:01.437544+0000","last_became_peered":"2023-11-24T06:08:01.437544+0000","last_unstale":"2023-11-24T06:08:01.445642+0000","last_undegraded":"2023-11-24T06:08:01.445642+0000","last_fullsized":"2023-11-24T06:08:01.445642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:42.556322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.736546+0000","last_change":"2023-11-24T06:08:00.421447+0000","last_active":"2023-11-24T06:08:01.736546+0000","last_peered":"2023-11-24T06:08:01.736546+0000","last_clean":"2023-11-24T06:08:01.736546+0000","last_became_active":"2023-11-24T06:08:00.421145+0000","last_became_peered":"2023-11-24T06:08:00.421145+0000","last_unstale":"2023-11-24T06:08:01.736546+0000","last_undegraded":"2023-11-24T06:08:01.736546+0000","last_fullsized":"2023-11-24T06:08:01.736546+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:29.195643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.430086+0000","last_change":"2023-11-24T06:08:01.427774+0000","last_active":"2023-11-24T06:08:01.430086+0000","last_peered":"2023-11-24T06:08:01.430086+0000","last_clean":"2023-11-24T06:08:01.430086+0000","last_became_active":"2023-11-24T06:08:01.427435+0000","last_became_peered":"2023-11-24T06:08:01.427435+0000","last_unstale":"2023-11-24T06:08:01.430086+0000","last_undegraded":"2023-11-24T06:08:01.430086+0000","last_fullsized":"2023-11-24T06:08:01.430086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:12.434134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425252+0000","last_change":"2023-11-24T06:08:00.424213+0000","last_active":"2023-11-24T06:08:01.425252+0000","last_peered":"2023-11-24T06:08:01.425252+0000","last_clean":"2023-11-24T06:08:01.425252+0000","last_became_active":"2023-11-24T06:08:00.423943+0000","last_became_peered":"2023-11-24T06:08:00.423943+0000","last_unstale":"2023-11-24T06:08:01.425252+0000","last_undegraded":"2023-11-24T06:08:01.425252+0000","last_fullsized":"2023-11-24T06:08:01.425252+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:15.532537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.422192+0000","last_change":"2023-11-24T06:08:01.420581+0000","last_active":"2023-11-24T06:08:01.422192+0000","last_peered":"2023-11-24T06:08:01.422192+0000","last_clean":"2023-11-24T06:08:01.422192+0000","last_became_active":"2023-11-24T06:08:01.420103+0000","last_became_peered":"2023-11-24T06:08:01.420103+0000","last_unstale":"2023-11-24T06:08:01.422192+0000","last_undegraded":"2023-11-24T06:08:01.422192+0000","last_fullsized":"2023-11-24T06:08:01.422192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:21:46.421251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.423958+0000","last_change":"2023-11-24T06:08:00.420593+0000","last_active":"2023-11-24T06:08:01.423958+0000","last_peered":"2023-11-24T06:08:01.423958+0000","last_clean":"2023-11-24T06:08:01.423958+0000","last_became_active":"2023-11-24T06:08:00.420286+0000","last_became_peered":"2023-11-24T06:08:00.420286+0000","last_unstale":"2023-11-24T06:08:01.423958+0000","last_undegraded":"2023-11-24T06:08:01.423958+0000","last_fullsized":"2023-11-24T06:08:01.423958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:28.644360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.432944+0000","last_change":"2023-11-24T06:08:01.427055+0000","last_active":"2023-11-24T06:08:01.432944+0000","last_peered":"2023-11-24T06:08:01.432944+0000","last_clean":"2023-11-24T06:08:01.432944+0000","last_became_active":"2023-11-24T06:08:01.426823+0000","last_became_peered":"2023-11-24T06:08:01.426823+0000","last_unstale":"2023-11-24T06:08:01.432944+0000","last_undegraded":"2023-11-24T06:08:01.432944+0000","last_fullsized":"2023-11-24T06:08:01.432944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:32.299407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425355+0000","last_change":"2023-11-24T06:08:00.421597+0000","last_active":"2023-11-24T06:08:01.425355+0000","last_peered":"2023-11-24T06:08:01.425355+0000","last_clean":"2023-11-24T06:08:01.425355+0000","last_became_active":"2023-11-24T06:08:00.421316+0000","last_became_peered":"2023-11-24T06:08:00.421316+0000","last_unstale":"2023-11-24T06:08:01.425355+0000","last_undegraded":"2023-11-24T06:08:01.425355+0000","last_fullsized":"2023-11-24T06:08:01.425355+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:17.143751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433264+0000","last_change":"2023-11-24T06:08:01.430666+0000","last_active":"2023-11-24T06:08:01.433264+0000","last_peered":"2023-11-24T06:08:01.433264+0000","last_clean":"2023-11-24T06:08:01.433264+0000","last_became_active":"2023-11-24T06:08:01.430401+0000","last_became_peered":"2023-11-24T06:08:01.430401+0000","last_unstale":"2023-11-24T06:08:01.433264+0000","last_undegraded":"2023-11-24T06:08:01.433264+0000","last_fullsized":"2023-11-24T06:08:01.433264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:14.157001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.736248+0000","last_change":"2023-11-24T06:08:00.424291+0000","last_active":"2023-11-24T06:08:01.736248+0000","last_peered":"2023-11-24T06:08:01.736248+0000","last_clean":"2023-11-24T06:08:01.736248+0000","last_became_active":"2023-11-24T06:08:00.422721+0000","last_became_peered":"2023-11-24T06:08:00.422721+0000","last_unstale":"2023-11-24T06:08:01.736248+0000","last_undegraded":"2023-11-24T06:08:01.736248+0000","last_fullsized":"2023-11-24T06:08:01.736248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:29.195643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433345+0000","last_change":"2023-11-24T06:08:01.424141+0000","last_active":"2023-11-24T06:08:01.433345+0000","last_peered":"2023-11-24T06:08:01.433345+0000","last_clean":"2023-11-24T06:08:01.433345+0000","last_became_active":"2023-11-24T06:08:01.423837+0000","last_became_peered":"2023-11-24T06:08:01.423837+0000","last_unstale":"2023-11-24T06:08:01.433345+0000","last_undegraded":"2023-11-24T06:08:01.433345+0000","last_fullsized":"2023-11-24T06:08:01.433345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:12.434134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.417729+0000","last_change":"2023-11-24T06:08:00.420556+0000","last_active":"2023-11-24T06:08:01.417729+0000","last_peered":"2023-11-24T06:08:01.417729+0000","last_clean":"2023-11-24T06:08:01.417729+0000","last_became_active":"2023-11-24T06:08:00.420250+0000","last_became_peered":"2023-11-24T06:08:00.420250+0000","last_unstale":"2023-11-24T06:08:01.417729+0000","last_undegraded":"2023-11-24T06:08:01.417729+0000","last_fullsized":"2023-11-24T06:08:01.417729+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:11.429196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.442315+0000","last_change":"2023-11-24T06:08:01.440217+0000","last_active":"2023-11-24T06:08:01.442315+0000","last_peered":"2023-11-24T06:08:01.442315+0000","last_clean":"2023-11-24T06:08:01.442315+0000","last_became_active":"2023-11-24T06:08:01.439942+0000","last_became_peered":"2023-11-24T06:08:01.439942+0000","last_unstale":"2023-11-24T06:08:01.442315+0000","last_undegraded":"2023-11-24T06:08:01.442315+0000","last_fullsized":"2023-11-24T06:08:01.442315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:52.422428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.738694+0000","last_change":"2023-11-24T06:08:00.413735+0000","last_active":"2023-11-24T06:08:01.738694+0000","last_peered":"2023-11-24T06:08:01.738694+0000","last_clean":"2023-11-24T06:08:01.738694+0000","last_became_active":"2023-11-24T06:08:00.413375+0000","last_became_peered":"2023-11-24T06:08:00.413375+0000","last_unstale":"2023-11-24T06:08:01.738694+0000","last_undegraded":"2023-11-24T06:08:01.738694+0000","last_fullsized":"2023-11-24T06:08:01.738694+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:20.079860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437726+0000","last_change":"2023-11-24T06:08:01.435772+0000","last_active":"2023-11-24T06:08:01.437726+0000","last_peered":"2023-11-24T06:08:01.437726+0000","last_clean":"2023-11-24T06:08:01.437726+0000","last_became_active":"2023-11-24T06:08:01.435495+0000","last_became_peered":"2023-11-24T06:08:01.435495+0000","last_unstale":"2023-11-24T06:08:01.437726+0000","last_undegraded":"2023-11-24T06:08:01.437726+0000","last_fullsized":"2023-11-24T06:08:01.437726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:30.200581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.738729+0000","last_change":"2023-11-24T06:08:00.415129+0000","last_active":"2023-11-24T06:08:01.738729+0000","last_peered":"2023-11-24T06:08:01.738729+0000","last_clean":"2023-11-24T06:08:01.738729+0000","last_became_active":"2023-11-24T06:08:00.414888+0000","last_became_peered":"2023-11-24T06:08:00.414888+0000","last_unstale":"2023-11-24T06:08:01.738729+0000","last_undegraded":"2023-11-24T06:08:01.738729+0000","last_fullsized":"2023-11-24T06:08:01.738729+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:41.551384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.443231+0000","last_change":"2023-11-24T06:08:01.442379+0000","last_active":"2023-11-24T06:08:01.443231+0000","last_peered":"2023-11-24T06:08:01.443231+0000","last_clean":"2023-11-24T06:08:01.443231+0000","last_became_active":"2023-11-24T06:08:01.442109+0000","last_became_peered":"2023-11-24T06:08:01.442109+0000","last_unstale":"2023-11-24T06:08:01.443231+0000","last_undegraded":"2023-11-24T06:08:01.443231+0000","last_fullsized":"2023-11-24T06:08:01.443231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:40.365444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433238+0000","last_change":"2023-11-24T06:08:01.429866+0000","last_active":"2023-11-24T06:08:01.433238+0000","last_peered":"2023-11-24T06:08:01.433238+0000","last_clean":"2023-11-24T06:08:01.433238+0000","last_became_active":"2023-11-24T06:08:01.429446+0000","last_became_peered":"2023-11-24T06:08:01.429446+0000","last_unstale":"2023-11-24T06:08:01.433238+0000","last_undegraded":"2023-11-24T06:08:01.433238+0000","last_fullsized":"2023-11-24T06:08:01.433238+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:14.157001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436208+0000","last_change":"2023-11-24T06:08:01.435709+0000","last_active":"2023-11-24T06:08:01.436208+0000","last_peered":"2023-11-24T06:08:01.436208+0000","last_clean":"2023-11-24T06:08:01.436208+0000","last_became_active":"2023-11-24T06:08:01.435488+0000","last_became_peered":"2023-11-24T06:08:01.435488+0000","last_unstale":"2023-11-24T06:08:01.436208+0000","last_undegraded":"2023-11-24T06:08:01.436208+0000","last_fullsized":"2023-11-24T06:08:01.436208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:14.157001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437910+0000","last_change":"2023-11-24T06:08:01.434372+0000","last_active":"2023-11-24T06:08:01.437910+0000","last_peered":"2023-11-24T06:08:01.437910+0000","last_clean":"2023-11-24T06:08:01.437910+0000","last_became_active":"2023-11-24T06:08:01.433776+0000","last_became_peered":"2023-11-24T06:08:01.433776+0000","last_unstale":"2023-11-24T06:08:01.437910+0000","last_undegraded":"2023-11-24T06:08:01.437910+0000","last_fullsized":"2023-11-24T06:08:01.437910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:12.434134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.435884+0000","last_change":"2023-11-24T06:08:01.433754+0000","last_active":"2023-11-24T06:08:01.435884+0000","last_peered":"2023-11-24T06:08:01.435884+0000","last_clean":"2023-11-24T06:08:01.435884+0000","last_became_active":"2023-11-24T06:08:01.433473+0000","last_became_peered":"2023-11-24T06:08:01.433473+0000","last_unstale":"2023-11-24T06:08:01.435884+0000","last_undegraded":"2023-11-24T06:08:01.435884+0000","last_fullsized":"2023-11-24T06:08:01.435884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:29.649298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433066+0000","last_change":"2023-11-24T06:08:01.428520+0000","last_active":"2023-11-24T06:08:01.433066+0000","last_peered":"2023-11-24T06:08:01.433066+0000","last_clean":"2023-11-24T06:08:01.433066+0000","last_became_active":"2023-11-24T06:08:01.428236+0000","last_became_peered":"2023-11-24T06:08:01.428236+0000","last_unstale":"2023-11-24T06:08:01.433066+0000","last_undegraded":"2023-11-24T06:08:01.433066+0000","last_fullsized":"2023-11-24T06:08:01.433066+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:16.537475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436403+0000","last_change":"2023-11-24T06:08:01.433972+0000","last_active":"2023-11-24T06:08:01.436403+0000","last_peered":"2023-11-24T06:08:01.436403+0000","last_clean":"2023-11-24T06:08:01.436403+0000","last_became_active":"2023-11-24T06:08:01.433685+0000","last_became_peered":"2023-11-24T06:08:01.433685+0000","last_unstale":"2023-11-24T06:08:01.436403+0000","last_undegraded":"2023-11-24T06:08:01.436403+0000","last_fullsized":"2023-11-24T06:08:01.436403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:16.537475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437386+0000","last_change":"2023-11-24T06:08:01.435802+0000","last_active":"2023-11-24T06:08:01.437386+0000","last_peered":"2023-11-24T06:08:01.437386+0000","last_clean":"2023-11-24T06:08:01.437386+0000","last_became_active":"2023-11-24T06:08:01.435228+0000","last_became_peered":"2023-11-24T06:08:01.435228+0000","last_unstale":"2023-11-24T06:08:01.437386+0000","last_undegraded":"2023-11-24T06:08:01.437386+0000","last_fullsized":"2023-11-24T06:08:01.437386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:37:39.086654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433575+0000","last_change":"2023-11-24T06:08:01.432295+0000","last_active":"2023-11-24T06:08:01.433575+0000","last_peered":"2023-11-24T06:08:01.433575+0000","last_clean":"2023-11-24T06:08:01.433575+0000","last_became_active":"2023-11-24T06:08:01.431767+0000","last_became_peered":"2023-11-24T06:08:01.431767+0000","last_unstale":"2023-11-24T06:08:01.433575+0000","last_undegraded":"2023-11-24T06:08:01.433575+0000","last_fullsized":"2023-11-24T06:08:01.433575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:18.148689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.437425+0000","last_change":"2023-11-24T06:08:01.434132+0000","last_active":"2023-11-24T06:08:01.437425+0000","last_peered":"2023-11-24T06:08:01.437425+0000","last_clean":"2023-11-24T06:08:01.437425+0000","last_became_active":"2023-11-24T06:08:01.433678+0000","last_became_peered":"2023-11-24T06:08:01.433678+0000","last_unstale":"2023-11-24T06:08:01.437425+0000","last_undegraded":"2023-11-24T06:08:01.437425+0000","last_fullsized":"2023-11-24T06:08:01.437425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:32.624231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.432819+0000","last_change":"2023-11-24T06:08:01.424409+0000","last_active":"2023-11-24T06:08:01.432819+0000","last_peered":"2023-11-24T06:08:01.432819+0000","last_clean":"2023-11-24T06:08:01.432819+0000","last_became_active":"2023-11-24T06:08:01.424152+0000","last_became_peered":"2023-11-24T06:08:01.424152+0000","last_unstale":"2023-11-24T06:08:01.432819+0000","last_undegraded":"2023-11-24T06:08:01.432819+0000","last_fullsized":"2023-11-24T06:08:01.432819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:32.624231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.438718+0000","last_change":"2023-11-24T06:08:01.436895+0000","last_active":"2023-11-24T06:08:01.438718+0000","last_peered":"2023-11-24T06:08:01.438718+0000","last_clean":"2023-11-24T06:08:01.438718+0000","last_became_active":"2023-11-24T06:08:01.436631+0000","last_became_peered":"2023-11-24T06:08:01.436631+0000","last_unstale":"2023-11-24T06:08:01.438718+0000","last_undegraded":"2023-11-24T06:08:01.438718+0000","last_fullsized":"2023-11-24T06:08:01.438718+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:32.299407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.436300+0000","last_change":"2023-11-24T06:08:01.435442+0000","last_active":"2023-11-24T06:08:01.436300+0000","last_peered":"2023-11-24T06:08:01.436300+0000","last_clean":"2023-11-24T06:08:01.436300+0000","last_became_active":"2023-11-24T06:08:01.435135+0000","last_became_peered":"2023-11-24T06:08:01.435135+0000","last_unstale":"2023-11-24T06:08:01.436300+0000","last_undegraded":"2023-11-24T06:08:01.436300+0000","last_fullsized":"2023-11-24T06:08:01.436300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:37:39.086654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.433058+0000","last_change":"2023-11-24T06:08:01.423313+0000","last_active":"2023-11-24T06:08:01.433058+0000","last_peered":"2023-11-24T06:08:01.433058+0000","last_clean":"2023-11-24T06:08:01.433058+0000","last_became_active":"2023-11-24T06:08:01.423042+0000","last_became_peered":"2023-11-24T06:08:01.423042+0000","last_unstale":"2023-11-24T06:08:01.433058+0000","last_undegraded":"2023-11-24T06:08:01.433058+0000","last_fullsized":"2023-11-24T06:08:01.433058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:32.299407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":655560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":57835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":200704,"data_stored":64770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":798720,"data_stored":655616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":66222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":64700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:08:06.213 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T06:08:06.214 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-24T06:08:06.214 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-24T06:08:06.214 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-24T06:08:06.214 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-24T06:08:06.369 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.365+0000 7f4d8ddef700 1 Processor -- start 2023-11-24T06:08:06.369 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.365+0000 7f4d8ddef700 1 -- start start 2023-11-24T06:08:06.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.366+0000 7f4d8ddef700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4d88143ae0 0x7f4d880b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:06.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.366+0000 7f4d8ddef700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4d881440f0 0x7f4d88071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:06.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.366+0000 7f4d8ddef700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d88143ec0 con 0x7f4d880720f0 2023-11-24T06:08:06.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.367+0000 7f4d8ddef700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d880b40a0 con 0x7f4d88143ae0 2023-11-24T06:08:06.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.367+0000 7f4d8ddef700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d88072560 con 0x7f4d881440f0 2023-11-24T06:08:06.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.367+0000 7f4d877fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4d88143ae0 0x7f4d880b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:06.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.367+0000 7f4d877fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4d88143ae0 0x7f4d880b51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57782/0 (socket says 172.21.15.91:57782) 2023-11-24T06:08:06.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.367+0000 7f4d877fe700 1 -- 172.21.15.91:0/1713339124 learned_addr learned my addr 172.21.15.91:0/1713339124 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:06.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.367+0000 7f4d86ffd700 1 --2- 172.21.15.91:0/1713339124 >> v2:172.21.15.94:3300/0 conn(0x7f4d881440f0 0x7f4d88071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:06.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.367+0000 7f4d877fe700 1 -- 172.21.15.91:0/1713339124 >> v2:172.21.15.94:3300/0 conn(0x7f4d881440f0 msgr2=0x7f4d88071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:06.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.367+0000 7f4d877fe700 1 --2- 172.21.15.91:0/1713339124 >> v2:172.21.15.94:3300/0 conn(0x7f4d881440f0 0x7f4d88071bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.367+0000 7f4d877fe700 1 -- 172.21.15.91:0/1713339124 >> v1:172.21.15.94:6789/0 conn(0x7f4d880720f0 legacy=0x7f4d8815e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:06.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.367+0000 7f4d877fe700 1 -- 172.21.15.91:0/1713339124 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d880b3ae0 con 0x7f4d88143ae0 2023-11-24T06:08:06.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.367+0000 7f4d877fe700 1 --2- 172.21.15.91:0/1713339124 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4d88143ae0 0x7f4d880b51b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4d880d1b00 tx=0x7f4d7c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d965d590f3bc5e66 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:06.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.367+0000 7f4d867fc700 1 -- 172.21.15.91:0/1713339124 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c004230 con 0x7f4d88143ae0 2023-11-24T06:08:06.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.367+0000 7f4d867fc700 1 -- 172.21.15.91:0/1713339124 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7c004390 con 0x7f4d88143ae0 2023-11-24T06:08:06.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.367+0000 7f4d867fc700 1 -- 172.21.15.91:0/1713339124 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c0036a0 con 0x7f4d88143ae0 2023-11-24T06:08:06.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.368+0000 7f4d8ddef700 1 -- 172.21.15.91:0/1713339124 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4d88143ae0 msgr2=0x7f4d880b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:06.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.368+0000 7f4d8ddef700 1 --2- 172.21.15.91:0/1713339124 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4d88143ae0 0x7f4d880b51b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4d880d1b00 tx=0x7f4d7c009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:06.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.368+0000 7f4d8ddef700 1 -- 172.21.15.91:0/1713339124 shutdown_connections 2023-11-24T06:08:06.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.368+0000 7f4d8ddef700 1 --2- 172.21.15.91:0/1713339124 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4d88143ae0 0x7f4d880b51b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.368+0000 7f4d8ddef700 1 --2- 172.21.15.91:0/1713339124 >> v2:172.21.15.94:3300/0 conn(0x7f4d881440f0 0x7f4d88071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.368+0000 7f4d8ddef700 1 -- 172.21.15.91:0/1713339124 >> 172.21.15.91:0/1713339124 conn(0x7f4d880b2740 msgr2=0x7f4d880b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:06.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.368+0000 7f4d8ddef700 1 -- 172.21.15.91:0/1713339124 shutdown_connections 2023-11-24T06:08:06.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.368+0000 7f4d8ddef700 1 -- 172.21.15.91:0/1713339124 wait complete. 2023-11-24T06:08:06.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.369+0000 7f4d8ddef700 1 Processor -- start 2023-11-24T06:08:06.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.369+0000 7f4d8ddef700 1 -- start start 2023-11-24T06:08:06.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d8ddef700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4d880720f0 0x7f4d881f0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:06.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d8ddef700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d88143ae0 0x7f4d880c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:06.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d8ddef700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d881440f0 0x7f4d880c6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:06.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d8ddef700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d880c7b40 con 0x7f4d880720f0 2023-11-24T06:08:06.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d8ddef700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d880c6f30 con 0x7f4d88143ae0 2023-11-24T06:08:06.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d8ddef700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d880c7210 con 0x7f4d881440f0 2023-11-24T06:08:06.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d877fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4d880720f0 0x7f4d881f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:06.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d877fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4d880720f0 0x7f4d881f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57788/0 (socket says 172.21.15.91:57788) 2023-11-24T06:08:06.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d877fe700 1 -- 172.21.15.91:0/2664815641 learned_addr learned my addr 172.21.15.91:0/2664815641 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:06.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d86ffd700 1 --2- 172.21.15.91:0/2664815641 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d88143ae0 0x7f4d880c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:06.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d87fff700 1 --2- 172.21.15.91:0/2664815641 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d881440f0 0x7f4d880c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:06.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d877fe700 1 -- 172.21.15.91:0/2664815641 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d881440f0 msgr2=0x7f4d880c6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:06.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d877fe700 1 --2- 172.21.15.91:0/2664815641 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d881440f0 0x7f4d880c6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d877fe700 1 -- 172.21.15.91:0/2664815641 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d88143ae0 msgr2=0x7f4d880c63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:06.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d877fe700 1 --2- 172.21.15.91:0/2664815641 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d88143ae0 0x7f4d880c63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d877fe700 1 -- 172.21.15.91:0/2664815641 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7c009550 con 0x7f4d880720f0 2023-11-24T06:08:06.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d87fff700 1 --2- 172.21.15.91:0/2664815641 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d881440f0 0x7f4d880c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:06.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.370+0000 7f4d877fe700 1 --2- 172.21.15.91:0/2664815641 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4d880720f0 0x7f4d881f0430 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009130 tx=0x7f4d7c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:06.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.371+0000 7f4d6ffff700 1 -- 172.21.15.91:0/2664815641 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c003bd0 con 0x7f4d880720f0 2023-11-24T06:08:06.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.371+0000 7f4d6ffff700 1 -- 172.21.15.91:0/2664815641 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7c020070 con 0x7f4d880720f0 2023-11-24T06:08:06.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.371+0000 7f4d8ddef700 1 -- 172.21.15.91:0/2664815641 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d880c74f0 con 0x7f4d880720f0 2023-11-24T06:08:06.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.371+0000 7f4d6ffff700 1 -- 172.21.15.91:0/2664815641 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c013840 con 0x7f4d880720f0 2023-11-24T06:08:06.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.371+0000 7f4d8ddef700 1 -- 172.21.15.91:0/2664815641 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d88157400 con 0x7f4d880720f0 2023-11-24T06:08:06.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.372+0000 7f4d8ddef700 1 -- 172.21.15.91:0/2664815641 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d88067a00 con 0x7f4d880720f0 2023-11-24T06:08:06.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.372+0000 7f4d6ffff700 1 -- 172.21.15.91:0/2664815641 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4d7c022030 con 0x7f4d880720f0 2023-11-24T06:08:06.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.372+0000 7f4d6ffff700 1 --2- 172.21.15.91:0/2664815641 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4d7406aed0 0x7f4d7406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:06.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.372+0000 7f4d86ffd700 1 --2- 172.21.15.91:0/2664815641 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4d7406aed0 0x7f4d7406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:06.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.373+0000 7f4d6ffff700 1 -- 172.21.15.91:0/2664815641 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d7c08cff0 con 0x7f4d880720f0 2023-11-24T06:08:06.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.373+0000 7f4d86ffd700 1 --2- 172.21.15.91:0/2664815641 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4d7406aed0 0x7f4d7406d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4d880b2710 tx=0x7f4d78008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:06.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.377+0000 7f4d6ffff700 1 -- 172.21.15.91:0/2664815641 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4d7c058130 con 0x7f4d880720f0 2023-11-24T06:08:06.627 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.623+0000 7f4d8ddef700 1 -- 172.21.15.91:0/2664815641 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f4d88067a00 con 0x7f4d880720f0 2023-11-24T06:08:06.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.626+0000 7f4d6ffff700 1 -- 172.21.15.91:0/2664815641 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171032 (secure 0 0 0) 0x7f4d7c05b750 con 0x7f4d880720f0 2023-11-24T06:08:06.632 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:06.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.629+0000 7f4d8ddef700 1 -- 172.21.15.91:0/2664815641 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4d7406aed0 msgr2=0x7f4d7406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:06.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.629+0000 7f4d8ddef700 1 --2- 172.21.15.91:0/2664815641 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4d7406aed0 0x7f4d7406d390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4d880b2710 tx=0x7f4d78008040 comp rx=0 tx=0).stop 2023-11-24T06:08:06.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.629+0000 7f4d8ddef700 1 -- 172.21.15.91:0/2664815641 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4d880720f0 msgr2=0x7f4d881f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:06.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.629+0000 7f4d8ddef700 1 --2- 172.21.15.91:0/2664815641 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4d880720f0 0x7f4d881f0430 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009130 tx=0x7f4d7c004750 comp rx=0 tx=0).stop 2023-11-24T06:08:06.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.629+0000 7f4d8ddef700 1 -- 172.21.15.91:0/2664815641 shutdown_connections 2023-11-24T06:08:06.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.629+0000 7f4d8ddef700 1 --2- 172.21.15.91:0/2664815641 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4d7406aed0 0x7f4d7406d390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.629+0000 7f4d8ddef700 1 --2- 172.21.15.91:0/2664815641 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4d881440f0 0x7f4d880c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.629+0000 7f4d8ddef700 1 --2- 172.21.15.91:0/2664815641 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4d880720f0 0x7f4d881f0430 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.629+0000 7f4d8ddef700 1 --2- 172.21.15.91:0/2664815641 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4d88143ae0 0x7f4d880c63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.629+0000 7f4d8ddef700 1 -- 172.21.15.91:0/2664815641 >> 172.21.15.91:0/2664815641 conn(0x7f4d880b2740 msgr2=0x7f4d88154340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:06.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.629+0000 7f4d8ddef700 1 -- 172.21.15.91:0/2664815641 shutdown_connections 2023-11-24T06:08:06.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.629+0000 7f4d8ddef700 1 -- 172.21.15.91:0/2664815641 wait complete. 2023-11-24T06:08:06.644 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":5,"active_gid":4107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6832","nonce":3377926323},{"type":"v1","addr":"172.21.15.91:6833","nonce":3377926323}]},"active_addr":"172.21.15.91:6833/3377926323","active_change":"2023-11-24T06:07:41.384549+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4103,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.91:0","nonce":3871997101}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.91:0","nonce":3558318390}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.91:0","nonce":3210152448}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.91:0","nonce":1543653275}]}]} 2023-11-24T06:08:06.647 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-24T06:08:06.648 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-24T06:08:06.648 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:06.769 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.765+0000 7f84b5092700 1 Processor -- start 2023-11-24T06:08:06.769 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.765+0000 7f84b5092700 1 -- start start 2023-11-24T06:08:06.771 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.765+0000 7f84b5092700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84b006b620 0x7f84b00b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:06.771 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.765+0000 7f84b5092700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f84b00b5ab0 0x7f84b00b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:06.771 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.765+0000 7f84b5092700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b0071f50 con 0x7f84b01478c0 2023-11-24T06:08:06.771 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.765+0000 7f84b5092700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b0072230 con 0x7f84b00b5ab0 2023-11-24T06:08:06.771 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.765+0000 7f84b5092700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b0072510 con 0x7f84b006b620 2023-11-24T06:08:06.771 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.766+0000 7f84ae59c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f84b00b5ab0 0x7f84b00b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:06.772 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.766+0000 7f84ae59c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f84b00b5ab0 0x7f84b00b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57796/0 (socket says 172.21.15.91:57796) 2023-11-24T06:08:06.772 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.766+0000 7f84ae59c700 1 -- 172.21.15.91:0/4223729848 learned_addr learned my addr 172.21.15.91:0/4223729848 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:06.772 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.766+0000 7f84aed9d700 1 --2- 172.21.15.91:0/4223729848 >> v2:172.21.15.94:3300/0 conn(0x7f84b006b620 0x7f84b00b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:06.772 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.766+0000 7f84ae59c700 1 -- 172.21.15.91:0/4223729848 >> v2:172.21.15.94:3300/0 conn(0x7f84b006b620 msgr2=0x7f84b00b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:06.773 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.766+0000 7f84ae59c700 1 --2- 172.21.15.91:0/4223729848 >> v2:172.21.15.94:3300/0 conn(0x7f84b006b620 0x7f84b00b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.773 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.766+0000 7f84ae59c700 1 -- 172.21.15.91:0/4223729848 >> v1:172.21.15.94:6789/0 conn(0x7f84b01478c0 legacy=0x7f84b0161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:06.773 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.766+0000 7f84ae59c700 1 -- 172.21.15.91:0/4223729848 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b00727f0 con 0x7f84b00b5ab0 2023-11-24T06:08:06.773 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.766+0000 7f84ae59c700 1 --2- 172.21.15.91:0/4223729848 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84b00b5ab0 0x7f84b00b3f80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f84b006d2e0 tx=0x7f84a0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a6804415c696d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:06.773 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.766+0000 7f84add9b700 1 -- 172.21.15.91:0/4223729848 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a000b870 con 0x7f84b00b5ab0 2023-11-24T06:08:06.773 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.766+0000 7f84add9b700 1 -- 172.21.15.91:0/4223729848 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a000b9d0 con 0x7f84b00b5ab0 2023-11-24T06:08:06.773 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.766+0000 7f84add9b700 1 -- 172.21.15.91:0/4223729848 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a000bd60 con 0x7f84b00b5ab0 2023-11-24T06:08:06.773 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.767+0000 7f84b5092700 1 -- 172.21.15.91:0/4223729848 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84b00b5ab0 msgr2=0x7f84b00b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:06.773 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.767+0000 7f84b5092700 1 --2- 172.21.15.91:0/4223729848 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84b00b5ab0 0x7f84b00b3f80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f84b006d2e0 tx=0x7f84a0009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.773 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.767+0000 7f84b5092700 1 -- 172.21.15.91:0/4223729848 shutdown_connections 2023-11-24T06:08:06.774 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.767+0000 7f84b5092700 1 --2- 172.21.15.91:0/4223729848 >> v2:172.21.15.94:3300/0 conn(0x7f84b006b620 0x7f84b00b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.774 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.767+0000 7f84b5092700 1 --2- 172.21.15.91:0/4223729848 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84b00b5ab0 0x7f84b00b3f80 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.774 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.767+0000 7f84b5092700 1 -- 172.21.15.91:0/4223729848 >> 172.21.15.91:0/4223729848 conn(0x7f84b00b2740 msgr2=0x7f84b00b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:06.774 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.767+0000 7f84b5092700 1 -- 172.21.15.91:0/4223729848 shutdown_connections 2023-11-24T06:08:06.774 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.767+0000 7f84b5092700 1 -- 172.21.15.91:0/4223729848 wait complete. 2023-11-24T06:08:06.776 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.768+0000 7f84b5092700 1 Processor -- start 2023-11-24T06:08:06.776 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.768+0000 7f84b5092700 1 -- start start 2023-11-24T06:08:06.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84b5092700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84b006b620 0x7f84b01f6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:06.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84b5092700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84b00b5ab0 0x7f84b01f7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:06.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84b5092700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84b01478c0 0x7f84b0138e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:06.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84b5092700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b00ca5c0 con 0x7f84b006b620 2023-11-24T06:08:06.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84b5092700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b013a430 con 0x7f84b00b5ab0 2023-11-24T06:08:06.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84b5092700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b013a710 con 0x7f84b01478c0 2023-11-24T06:08:06.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84aed9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84b006b620 0x7f84b01f6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:06.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84aed9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84b006b620 0x7f84b01f6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57802/0 (socket says 172.21.15.91:57802) 2023-11-24T06:08:06.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84aed9d700 1 -- 172.21.15.91:0/1327093659 learned_addr learned my addr 172.21.15.91:0/1327093659 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:06.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84ae59c700 1 --2- 172.21.15.91:0/1327093659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84b00b5ab0 0x7f84b01f7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:06.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84af59e700 1 --2- 172.21.15.91:0/1327093659 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84b01478c0 0x7f84b0138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:06.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84aed9d700 1 -- 172.21.15.91:0/1327093659 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84b01478c0 msgr2=0x7f84b0138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:06.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84aed9d700 1 --2- 172.21.15.91:0/1327093659 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84b01478c0 0x7f84b0138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84aed9d700 1 -- 172.21.15.91:0/1327093659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84b00b5ab0 msgr2=0x7f84b01f7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:06.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84aed9d700 1 --2- 172.21.15.91:0/1327093659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84b00b5ab0 0x7f84b01f7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84aed9d700 1 -- 172.21.15.91:0/1327093659 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a0009550 con 0x7f84b006b620 2023-11-24T06:08:06.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.769+0000 7f84aed9d700 1 --2- 172.21.15.91:0/1327093659 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84b006b620 0x7f84b01f6fe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f84a400e810 tx=0x7f84a400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:06.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.770+0000 7f84977fe700 1 -- 172.21.15.91:0/1327093659 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a400c070 con 0x7f84b006b620 2023-11-24T06:08:06.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.770+0000 7f84977fe700 1 -- 172.21.15.91:0/1327093659 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a4013610 con 0x7f84b006b620 2023-11-24T06:08:06.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.770+0000 7f84b5092700 1 -- 172.21.15.91:0/1327093659 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b013a9f0 con 0x7f84b006b620 2023-11-24T06:08:06.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.770+0000 7f84977fe700 1 -- 172.21.15.91:0/1327093659 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a40139c0 con 0x7f84b006b620 2023-11-24T06:08:06.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.770+0000 7f84b5092700 1 -- 172.21.15.91:0/1327093659 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b0139370 con 0x7f84b006b620 2023-11-24T06:08:06.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.771+0000 7f84b5092700 1 -- 172.21.15.91:0/1327093659 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b0068670 con 0x7f84b006b620 2023-11-24T06:08:06.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.771+0000 7f84977fe700 1 -- 172.21.15.91:0/1327093659 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f84a4013b20 con 0x7f84b006b620 2023-11-24T06:08:06.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.771+0000 7f84977fe700 1 --2- 172.21.15.91:0/1327093659 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f849c06aed0 0x7f849c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:06.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.771+0000 7f84ae59c700 1 --2- 172.21.15.91:0/1327093659 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f849c06aed0 0x7f849c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:06.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.771+0000 7f84977fe700 1 -- 172.21.15.91:0/1327093659 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84a40892e0 con 0x7f84b006b620 2023-11-24T06:08:06.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.772+0000 7f84ae59c700 1 --2- 172.21.15.91:0/1327093659 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f849c06aed0 0x7f849c06d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f84b00b5f10 tx=0x7f84a0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:06.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.776+0000 7f84977fe700 1 -- 172.21.15.91:0/1327093659 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f84a40543a0 con 0x7f84b006b620 2023-11-24T06:08:06.970 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.966+0000 7f84b5092700 1 -- 172.21.15.91:0/1327093659 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84b006bd40 con 0x7f84b006b620 2023-11-24T06:08:06.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.967+0000 7f84977fe700 1 -- 172.21.15.91:0/1327093659 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7f84a40579c0 con 0x7f84b006b620 2023-11-24T06:08:06.971 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:06.972 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":20,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:02.415612+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:06.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.969+0000 7f84b5092700 1 -- 172.21.15.91:0/1327093659 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f849c06aed0 msgr2=0x7f849c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:06.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.969+0000 7f84b5092700 1 --2- 172.21.15.91:0/1327093659 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f849c06aed0 0x7f849c06d390 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f84b00b5f10 tx=0x7f84a0008030 comp rx=0 tx=0).stop 2023-11-24T06:08:06.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.969+0000 7f84b5092700 1 -- 172.21.15.91:0/1327093659 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84b006b620 msgr2=0x7f84b01f6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:06.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.969+0000 7f84b5092700 1 --2- 172.21.15.91:0/1327093659 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84b006b620 0x7f84b01f6fe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f84a400e810 tx=0x7f84a400eb20 comp rx=0 tx=0).stop 2023-11-24T06:08:06.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.969+0000 7f84b5092700 1 -- 172.21.15.91:0/1327093659 shutdown_connections 2023-11-24T06:08:06.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.969+0000 7f84b5092700 1 --2- 172.21.15.91:0/1327093659 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f849c06aed0 0x7f849c06d390 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.969+0000 7f84b5092700 1 --2- 172.21.15.91:0/1327093659 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84b01478c0 0x7f84b0138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.969+0000 7f84b5092700 1 --2- 172.21.15.91:0/1327093659 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84b006b620 0x7f84b01f6fe0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.969+0000 7f84b5092700 1 --2- 172.21.15.91:0/1327093659 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84b00b5ab0 0x7f84b01f7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:06.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.969+0000 7f84b5092700 1 -- 172.21.15.91:0/1327093659 >> 172.21.15.91:0/1327093659 conn(0x7f84b00b2740 msgr2=0x7f84b00b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:06.975 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.970+0000 7f84b5092700 1 -- 172.21.15.91:0/1327093659 shutdown_connections 2023-11-24T06:08:06.975 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:06.970+0000 7f84b5092700 1 -- 172.21.15.91:0/1327093659 wait complete. 2023-11-24T06:08:06.986 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-24T06:08:06.986 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:07.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.102+0000 7f4ca8261700 1 Processor -- start 2023-11-24T06:08:07.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.102+0000 7f4ca8261700 1 -- start start 2023-11-24T06:08:07.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.103+0000 7f4ca8261700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4ca01446a0 0x7f4ca00b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.103+0000 7f4ca8261700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4ca006d810 0x7f4ca00cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.103+0000 7f4ca8261700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca0144a80 con 0x7f4ca015e8b0 2023-11-24T06:08:07.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.103+0000 7f4ca8261700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca006dfb0 con 0x7f4ca01446a0 2023-11-24T06:08:07.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.103+0000 7f4ca8261700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca006e290 con 0x7f4ca006d810 2023-11-24T06:08:07.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.103+0000 7f4ca5ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4ca01446a0 0x7f4ca00b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.103+0000 7f4ca67fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f4ca015e8b0 0x7f4ca006f050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52746/0 (socket says 172.21.15.91:52746) 2023-11-24T06:08:07.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.103+0000 7f4ca67fe700 1 -- 172.21.15.91:0/1568396706 learned_addr learned my addr 172.21.15.91:0/1568396706 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:07.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.103+0000 7f4ca57fc700 1 --2- 172.21.15.91:0/1568396706 >> v2:172.21.15.94:3300/0 conn(0x7f4ca006d810 0x7f4ca00cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.103+0000 7f4ca4ffb700 1 -- 172.21.15.91:0/1568396706 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623628825 0 0) 0x7f4ca0144a80 con 0x7f4ca015e8b0 2023-11-24T06:08:07.110 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.103+0000 7f4ca4ffb700 1 -- 172.21.15.91:0/1568396706 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c8c0035b0 con 0x7f4ca015e8b0 2023-11-24T06:08:07.110 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.103+0000 7f4ca5ffd700 1 -- 172.21.15.91:0/1568396706 >> v2:172.21.15.94:3300/0 conn(0x7f4ca006d810 msgr2=0x7f4ca00cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.110 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.103+0000 7f4ca5ffd700 1 --2- 172.21.15.91:0/1568396706 >> v2:172.21.15.94:3300/0 conn(0x7f4ca006d810 0x7f4ca00cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.110 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.103+0000 7f4ca5ffd700 1 -- 172.21.15.91:0/1568396706 >> v1:172.21.15.91:6789/0 conn(0x7f4ca015e8b0 legacy=0x7f4ca006f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.110 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.103+0000 7f4ca5ffd700 1 -- 172.21.15.91:0/1568396706 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca006e570 con 0x7f4ca01446a0 2023-11-24T06:08:07.110 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.104+0000 7f4ca5ffd700 1 --2- 172.21.15.91:0/1568396706 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ca01446a0 0x7f4ca00b5700 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00b26d0 tx=0x7f4c90009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d19ed7a4e0472c34 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.110 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.104+0000 7f4ca4ffb700 1 -- 172.21.15.91:0/1568396706 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c90011830 con 0x7f4ca01446a0 2023-11-24T06:08:07.110 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.104+0000 7f4ca4ffb700 1 -- 172.21.15.91:0/1568396706 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c90011990 con 0x7f4ca01446a0 2023-11-24T06:08:07.110 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.104+0000 7f4ca4ffb700 1 -- 172.21.15.91:0/1568396706 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c90011d20 con 0x7f4ca01446a0 2023-11-24T06:08:07.110 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.104+0000 7f4ca8261700 1 -- 172.21.15.91:0/1568396706 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ca01446a0 msgr2=0x7f4ca00b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.110 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.104+0000 7f4ca8261700 1 --2- 172.21.15.91:0/1568396706 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ca01446a0 0x7f4ca00b5700 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4ca00b26d0 tx=0x7f4c90009b20 comp rx=0 tx=0).stop 2023-11-24T06:08:07.111 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.104+0000 7f4ca8261700 1 -- 172.21.15.91:0/1568396706 shutdown_connections 2023-11-24T06:08:07.111 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.104+0000 7f4ca8261700 1 --2- 172.21.15.91:0/1568396706 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ca01446a0 0x7f4ca00b5700 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.111 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.104+0000 7f4ca8261700 1 --2- 172.21.15.91:0/1568396706 >> v2:172.21.15.94:3300/0 conn(0x7f4ca006d810 0x7f4ca00cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.111 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.104+0000 7f4ca8261700 1 -- 172.21.15.91:0/1568396706 >> 172.21.15.91:0/1568396706 conn(0x7f4ca00d0ad0 msgr2=0x7f4ca00d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.111 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.104+0000 7f4ca8261700 1 -- 172.21.15.91:0/1568396706 shutdown_connections 2023-11-24T06:08:07.111 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.104+0000 7f4ca8261700 1 -- 172.21.15.91:0/1568396706 wait complete. 2023-11-24T06:08:07.114 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.106+0000 7f4ca8261700 1 Processor -- start 2023-11-24T06:08:07.114 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.106+0000 7f4ca8261700 1 -- start start 2023-11-24T06:08:07.114 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.106+0000 7f4ca8261700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ca006d810 0x7f4ca0149b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.114 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.106+0000 7f4ca8261700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ca01446a0 0x7f4ca013a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.114 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.106+0000 7f4ca8261700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ca015e8b0 0x7f4ca015d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.114 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.106+0000 7f4ca8261700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca014a060 con 0x7f4ca01446a0 2023-11-24T06:08:07.114 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.106+0000 7f4ca8261700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca01f5c60 con 0x7f4ca006d810 2023-11-24T06:08:07.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.106+0000 7f4ca8261700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca01f50e0 con 0x7f4ca015e8b0 2023-11-24T06:08:07.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.106+0000 7f4ca57fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ca01446a0 0x7f4ca013a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.106+0000 7f4ca57fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ca01446a0 0x7f4ca013a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57816/0 (socket says 172.21.15.91:57816) 2023-11-24T06:08:07.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.106+0000 7f4ca57fc700 1 -- 172.21.15.91:0/2265810559 learned_addr learned my addr 172.21.15.91:0/2265810559 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.106+0000 7f4ca5ffd700 1 --2- 172.21.15.91:0/2265810559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ca006d810 0x7f4ca0149b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.107+0000 7f4ca67fe700 1 --2- 172.21.15.91:0/2265810559 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ca015e8b0 0x7f4ca015d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.107+0000 7f4ca57fc700 1 -- 172.21.15.91:0/2265810559 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ca015e8b0 msgr2=0x7f4ca015d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.107+0000 7f4ca57fc700 1 --2- 172.21.15.91:0/2265810559 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ca015e8b0 0x7f4ca015d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.107+0000 7f4ca57fc700 1 -- 172.21.15.91:0/2265810559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ca006d810 msgr2=0x7f4ca0149b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.107+0000 7f4ca57fc700 1 --2- 172.21.15.91:0/2265810559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ca006d810 0x7f4ca0149b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.107+0000 7f4ca57fc700 1 -- 172.21.15.91:0/2265810559 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c900094a0 con 0x7f4ca01446a0 2023-11-24T06:08:07.116 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.107+0000 7f4ca57fc700 1 --2- 172.21.15.91:0/2265810559 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ca01446a0 0x7f4ca013a400 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00e870 tx=0x7f4c9c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.116 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.107+0000 7f4c967fc700 1 -- 172.21.15.91:0/2265810559 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c009590 con 0x7f4ca01446a0 2023-11-24T06:08:07.116 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.107+0000 7f4c967fc700 1 -- 172.21.15.91:0/2265810559 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c9c013680 con 0x7f4ca01446a0 2023-11-24T06:08:07.116 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.107+0000 7f4c967fc700 1 -- 172.21.15.91:0/2265810559 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c0139f0 con 0x7f4ca01446a0 2023-11-24T06:08:07.116 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.107+0000 7f4ca8261700 1 -- 172.21.15.91:0/2265810559 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca01f53c0 con 0x7f4ca01446a0 2023-11-24T06:08:07.116 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.107+0000 7f4ca8261700 1 -- 172.21.15.91:0/2265810559 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca01f5980 con 0x7f4ca01446a0 2023-11-24T06:08:07.116 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.108+0000 7f4ca8261700 1 -- 172.21.15.91:0/2265810559 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca0067f60 con 0x7f4ca01446a0 2023-11-24T06:08:07.116 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.108+0000 7f4c967fc700 1 -- 172.21.15.91:0/2265810559 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4c9c013b50 con 0x7f4ca01446a0 2023-11-24T06:08:07.116 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.109+0000 7f4c967fc700 1 --2- 172.21.15.91:0/2265810559 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c8406aed0 0x7f4c8406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.116 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.109+0000 7f4c967fc700 1 -- 172.21.15.91:0/2265810559 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c9c0894b0 con 0x7f4ca01446a0 2023-11-24T06:08:07.116 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.109+0000 7f4ca5ffd700 1 --2- 172.21.15.91:0/2265810559 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c8406aed0 0x7f4c8406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.116 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.109+0000 7f4ca5ffd700 1 --2- 172.21.15.91:0/2265810559 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c8406aed0 0x7f4c8406d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4c900090b0 tx=0x7f4c90008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.113+0000 7f4c967fc700 1 -- 172.21.15.91:0/2265810559 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c9c0545f0 con 0x7f4ca01446a0 2023-11-24T06:08:07.308 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.304+0000 7f4ca8261700 1 -- 172.21.15.91:0/2265810559 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ca00d16f0 con 0x7f4ca01446a0 2023-11-24T06:08:07.309 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.305+0000 7f4c967fc700 1 -- 172.21.15.91:0/2265810559 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7f4c9c057c10 con 0x7f4ca01446a0 2023-11-24T06:08:07.309 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:07.309 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":20,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:02.415612+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:07.311 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.307+0000 7f4ca8261700 1 -- 172.21.15.91:0/2265810559 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c8406aed0 msgr2=0x7f4c8406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.311 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.307+0000 7f4ca8261700 1 --2- 172.21.15.91:0/2265810559 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c8406aed0 0x7f4c8406d390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4c900090b0 tx=0x7f4c90008030 comp rx=0 tx=0).stop 2023-11-24T06:08:07.311 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.307+0000 7f4ca8261700 1 -- 172.21.15.91:0/2265810559 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ca01446a0 msgr2=0x7f4ca013a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.311 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.307+0000 7f4ca8261700 1 --2- 172.21.15.91:0/2265810559 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ca01446a0 0x7f4ca013a400 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c00e870 tx=0x7f4c9c00eb80 comp rx=0 tx=0).stop 2023-11-24T06:08:07.312 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.307+0000 7f4ca8261700 1 -- 172.21.15.91:0/2265810559 shutdown_connections 2023-11-24T06:08:07.312 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.307+0000 7f4ca8261700 1 --2- 172.21.15.91:0/2265810559 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c8406aed0 0x7f4c8406d390 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.312 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.307+0000 7f4ca8261700 1 --2- 172.21.15.91:0/2265810559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ca006d810 0x7f4ca0149b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.312 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.307+0000 7f4ca8261700 1 --2- 172.21.15.91:0/2265810559 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ca015e8b0 0x7f4ca015d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.312 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.307+0000 7f4ca8261700 1 --2- 172.21.15.91:0/2265810559 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ca01446a0 0x7f4ca013a400 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.312 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.307+0000 7f4ca8261700 1 -- 172.21.15.91:0/2265810559 >> 172.21.15.91:0/2265810559 conn(0x7f4ca00d0ad0 msgr2=0x7f4ca00b4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.312 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.307+0000 7f4ca8261700 1 -- 172.21.15.91:0/2265810559 shutdown_connections 2023-11-24T06:08:07.312 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.307+0000 7f4ca8261700 1 -- 172.21.15.91:0/2265810559 wait complete. 2023-11-24T06:08:07.324 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-24T06:08:07.324 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-24T06:08:07.324 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-24T06:08:07.325 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-11-24T06:08:07.325 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-11-24T06:08:07.325 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-11-24T06:08:07.326 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-11-24T06:08:07.326 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-11-24T06:08:07.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.531+0000 7fe2fefeb700 1 Processor -- start 2023-11-24T06:08:07.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.532+0000 7fe2fefeb700 1 -- start start 2023-11-24T06:08:07.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.532+0000 7fe2fefeb700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe2f8145870 0x7fe2f8165ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.532+0000 7fe2fefeb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe2f8166430 0x7fe2f80cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.532+0000 7fe2fefeb700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f80ca8e0 con 0x7fe2f8070a80 2023-11-24T06:08:07.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.532+0000 7fe2fefeb700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2f80c9fd0 con 0x7fe2f8145870 2023-11-24T06:08:07.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.532+0000 7fe2fefeb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f8166c60 con 0x7fe2f8166430 2023-11-24T06:08:07.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.533+0000 7fe2effff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe2f8145870 0x7fe2f8165ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.533+0000 7fe2effff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe2f8145870 0x7fe2f8165ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:42214/0 (socket says 172.21.15.91:42214) 2023-11-24T06:08:07.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.533+0000 7fe2effff700 1 -- 172.21.15.91:0/3821111437 learned_addr learned my addr 172.21.15.91:0/3821111437 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.533+0000 7fe2effff700 1 -- 172.21.15.91:0/3821111437 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe2f8166430 msgr2=0x7fe2f80cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.533+0000 7fe2effff700 1 --2- 172.21.15.91:0/3821111437 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe2f8166430 0x7fe2f80cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.533+0000 7fe2effff700 1 -- 172.21.15.91:0/3821111437 >> v1:172.21.15.94:6789/0 conn(0x7fe2f8070a80 legacy=0x7fe2f8070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.533+0000 7fe2effff700 1 -- 172.21.15.91:0/3821111437 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f80ca290 con 0x7fe2f8145870 2023-11-24T06:08:07.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.534+0000 7f2d62b27700 1 Processor -- start 2023-11-24T06:08:07.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.534+0000 7fe2effff700 1 --2- 172.21.15.91:0/3821111437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2f8145870 0x7fe2f8165ef0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe2e80098f0 tx=0x7fe2e8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39d0cac80e1fb46a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.534+0000 7f2d62b27700 1 -- start start 2023-11-24T06:08:07.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.535+0000 7f8d82f04700 1 Processor -- start 2023-11-24T06:08:07.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.535+0000 7f8d82f04700 1 -- start start 2023-11-24T06:08:07.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.536+0000 7f2d62b27700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f2d5c0d04a0 0x7f2d5c15ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.536+0000 7f2d62b27700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2d5c15b170 0x7f2d5c15db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.536+0000 7f2d62b27700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d5c0ce000 con 0x7f2d5c071390 2023-11-24T06:08:07.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.536+0000 7f2d62b27700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d5c0cb2d0 con 0x7f2d5c0d04a0 2023-11-24T06:08:07.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.536+0000 7f2d62b27700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d5c0ce2c0 con 0x7f2d5c15b170 2023-11-24T06:08:07.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.535+0000 7f9ef8f26700 1 Processor -- start 2023-11-24T06:08:07.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.535+0000 7f9ef8f26700 1 -- start start 2023-11-24T06:08:07.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.536+0000 7f2d610c4700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2d5c15b170 0x7f2d5c15db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.536+0000 7f2d610c4700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2d5c15b170 0x7f2d5c15db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:42222/0 (socket says 172.21.15.91:42222) 2023-11-24T06:08:07.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.536+0000 7f2d610c4700 1 -- 172.21.15.91:0/4132462276 learned_addr learned my addr 172.21.15.91:0/4132462276 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.536+0000 7f8d82f04700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f8d7c06f0c0 0x7f8d7c070e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.536+0000 7f8d82f04700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d7c071390 0x7f8d7c0cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.536+0000 7f8d82f04700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7c0cd550 con 0x7f8d7c145700 2023-11-24T06:08:07.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.536+0000 7f8d82f04700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7c071bc0 con 0x7f8d7c06f0c0 2023-11-24T06:08:07.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.536+0000 7f8d82f04700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c071ea0 con 0x7f8d7c071390 2023-11-24T06:08:07.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f2d610c4700 1 -- 172.21.15.91:0/4132462276 >> v2:172.21.15.91:3300/0 conn(0x7f2d5c0d04a0 msgr2=0x7f2d5c15ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f2d610c4700 1 --2- 172.21.15.91:0/4132462276 >> v2:172.21.15.91:3300/0 conn(0x7f2d5c0d04a0 0x7f2d5c15ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f2d610c4700 1 -- 172.21.15.91:0/4132462276 >> v1:172.21.15.91:6789/0 conn(0x7f2d5c071390 legacy=0x7f2d5c15a520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-24T06:08:07.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f2d610c4700 1 -- 172.21.15.91:0/4132462276 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5c0ce5a0 con 0x7f2d5c15b170 2023-11-24T06:08:07.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f2d610c4700 1 --2- 172.21.15.91:0/4132462276 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d5c15b170 0x7f2d5c15db10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c0098f0 tx=0x7f2d4c009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b62888f765d10615 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f9ef8f26700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f9ef415a4c0 0x7f9ef406d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f9ef8f26700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef40cded0 con 0x7f9ef40c9fd0 2023-11-24T06:08:07.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f9ef8f26700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef40ce1b0 con 0x7f9ef40cef70 2023-11-24T06:08:07.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f9ef8f26700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ef40ce490 con 0x7f9ef415a4c0 2023-11-24T06:08:07.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f9ef259c700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f9ef40c9fd0 0x7f9ef40ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52772/0 (socket says 172.21.15.91:52772) 2023-11-24T06:08:07.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f9ef1d9b700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f9ef40cef70 0x7f9ef406cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.91:54566/0 (socket says 172.21.15.91:54566) 2023-11-24T06:08:07.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f9ef259c700 1 -- 172.21.15.91:0/1258664499 learned_addr learned my addr 172.21.15.91:0/1258664499 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:07.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f2d5b7fe700 1 -- 172.21.15.91:0/4132462276 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c004230 con 0x7f2d5c15b170 2023-11-24T06:08:07.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f2d5b7fe700 1 -- 172.21.15.91:0/4132462276 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4c004390 con 0x7f2d5c15b170 2023-11-24T06:08:07.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f2d5b7fe700 1 -- 172.21.15.91:0/4132462276 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c0036a0 con 0x7f2d5c15b170 2023-11-24T06:08:07.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f9ef2d9d700 1 --2- 172.21.15.91:0/1258664499 >> v2:172.21.15.91:3300/0 conn(0x7f9ef415a4c0 0x7f9ef406d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f9ef159a700 1 -- 172.21.15.91:0/1258664499 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893494114 0 0) 0x7f9ef40ce1b0 con 0x7f9ef40cef70 2023-11-24T06:08:07.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.537+0000 7f9ef159a700 1 -- 172.21.15.91:0/1258664499 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9edc0035b0 con 0x7f9ef40cef70 2023-11-24T06:08:07.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef159a700 1 -- 172.21.15.91:0/1258664499 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2856451286 0 0) 0x7f9edc0035b0 con 0x7f9ef40cef70 2023-11-24T06:08:07.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef159a700 1 -- 172.21.15.91:0/1258664499 >> v2:172.21.15.91:3300/0 conn(0x7f9ef415a4c0 msgr2=0x7f9ef406d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef159a700 1 --2- 172.21.15.91:0/1258664499 >> v2:172.21.15.91:3300/0 conn(0x7f9ef415a4c0 0x7f9ef406d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef159a700 1 -- 172.21.15.91:0/1258664499 >> v1:172.21.15.91:6789/0 conn(0x7f9ef40c9fd0 legacy=0x7f9ef40ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef159a700 1 -- 172.21.15.91:0/1258664499 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef40ce770 con 0x7f9ef40cef70 2023-11-24T06:08:07.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef159a700 1 -- 172.21.15.91:0/1258664499 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f9ee4000ed0 con 0x7f9ef40cef70 2023-11-24T06:08:07.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef159a700 1 -- 172.21.15.91:0/1258664499 >> v1:172.21.15.94:6789/0 conn(0x7f9ef40cef70 legacy=0x7f9ef406cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef159a700 1 --2- 172.21.15.91:0/1258664499 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9edc003a00 0x7f9edc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef159a700 1 --2- 172.21.15.91:0/1258664499 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9edc004850 0x7f9edc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef159a700 1 --2- 172.21.15.91:0/1258664499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc005210 0x7f9edc00dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef159a700 1 -- 172.21.15.91:0/1258664499 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef40ce490 con 0x7f9edc003a00 2023-11-24T06:08:07.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef159a700 1 -- 172.21.15.91:0/1258664499 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc00e2c0 con 0x7f9edc005210 2023-11-24T06:08:07.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef159a700 1 -- 172.21.15.91:0/1258664499 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc0059e0 con 0x7f9edc004850 2023-11-24T06:08:07.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef2d9d700 1 --2- 172.21.15.91:0/1258664499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc005210 0x7f9edc00dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef1d9b700 1 --2- 172.21.15.91:0/1258664499 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9edc004850 0x7f9edc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef259c700 1 --2- 172.21.15.91:0/1258664499 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9edc003a00 0x7f9edc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef2d9d700 1 -- 172.21.15.91:0/1258664499 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9edc004850 msgr2=0x7f9edc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef2d9d700 1 --2- 172.21.15.91:0/1258664499 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9edc004850 0x7f9edc004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef2d9d700 1 -- 172.21.15.91:0/1258664499 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9edc003a00 msgr2=0x7f9edc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef2d9d700 1 --2- 172.21.15.91:0/1258664499 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9edc003a00 0x7f9edc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef2d9d700 1 -- 172.21.15.91:0/1258664499 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9edc0035b0 con 0x7f9edc005210 2023-11-24T06:08:07.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f8d80ca0700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f8d7c145700 0x7f8d7c06e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52766/0 (socket says 172.21.15.91:52766) 2023-11-24T06:08:07.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f8d80ca0700 1 -- 172.21.15.91:0/3175778384 learned_addr learned my addr 172.21.15.91:0/3175778384 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:07.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f8d7bfff700 1 --2- 172.21.15.91:0/3175778384 >> v2:172.21.15.91:3300/0 conn(0x7f8d7c06f0c0 0x7f8d7c070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f2d62b27700 1 -- 172.21.15.91:0/4132462276 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d5c15b170 msgr2=0x7f2d5c15db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f2d62b27700 1 --2- 172.21.15.91:0/4132462276 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d5c15b170 0x7f2d5c15db10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c0098f0 tx=0x7f2d4c009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:07.550 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef2d9d700 1 -- 172.21.15.91:0/1258664499 reap_dead start 2023-11-24T06:08:07.550 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.538+0000 7f9ef2d9d700 1 --2- 172.21.15.91:0/1258664499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc005210 0x7f9edc00dc20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9eec014a00 tx=0x7f9edc005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2924e9d3c8537f23 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.550 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7f2d62b27700 1 -- 172.21.15.91:0/4132462276 shutdown_connections 2023-11-24T06:08:07.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7f2d62b27700 1 --2- 172.21.15.91:0/4132462276 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d5c15b170 0x7f2d5c15db10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7f2d62b27700 1 --2- 172.21.15.91:0/4132462276 >> v2:172.21.15.91:3300/0 conn(0x7f2d5c0d04a0 0x7f2d5c15ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7f2d62b27700 1 -- 172.21.15.91:0/4132462276 >> 172.21.15.91:0/4132462276 conn(0x7f2d5c0cf980 msgr2=0x7f2d5c0cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7fe2ef7fe700 1 -- 172.21.15.91:0/3821111437 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8004230 con 0x7fe2f8145870 2023-11-24T06:08:07.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7fe2ef7fe700 1 -- 172.21.15.91:0/3821111437 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e8004390 con 0x7fe2f8145870 2023-11-24T06:08:07.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7fe2ef7fe700 1 -- 172.21.15.91:0/3821111437 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e80036a0 con 0x7fe2f8145870 2023-11-24T06:08:07.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.540+0000 7fe2fefeb700 1 -- 172.21.15.91:0/3821111437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2f8145870 msgr2=0x7fe2f8165ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.540+0000 7fe2fefeb700 1 --2- 172.21.15.91:0/3821111437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2f8145870 0x7fe2f8165ef0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe2e80098f0 tx=0x7fe2e8009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:07.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7f9ef159a700 1 -- 172.21.15.91:0/1258664499 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec019070 con 0x7f9edc005210 2023-11-24T06:08:07.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7f9ef159a700 1 -- 172.21.15.91:0/1258664499 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eec003a20 con 0x7f9edc005210 2023-11-24T06:08:07.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7f9ef159a700 1 -- 172.21.15.91:0/1258664499 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec003d90 con 0x7f9edc005210 2023-11-24T06:08:07.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7f8d7b7fe700 1 -- 172.21.15.91:0/3175778384 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340185149 0 0) 0x7f8d7c0cd550 con 0x7f8d7c145700 2023-11-24T06:08:07.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7f8d7b7fe700 1 -- 172.21.15.91:0/3175778384 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d6c0035b0 con 0x7f8d7c145700 2023-11-24T06:08:07.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7f8d7bfff700 1 -- 172.21.15.91:0/3175778384 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d7c071390 msgr2=0x7f8d7c0cd9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T06:08:07.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7f8d7bfff700 1 --2- 172.21.15.91:0/3175778384 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d7c071390 0x7f8d7c0cd9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7f8d7bfff700 1 -- 172.21.15.91:0/3175778384 >> v1:172.21.15.91:6789/0 conn(0x7f8d7c145700 legacy=0x7f8d7c06e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7f8d7bfff700 1 -- 172.21.15.91:0/3175778384 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c072180 con 0x7f8d7c06f0c0 2023-11-24T06:08:07.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.539+0000 7f8d7bfff700 1 --2- 172.21.15.91:0/3175778384 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d7c06f0c0 0x7f8d7c070e50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009840 tx=0x7f8d70009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf87a075c45395c3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.540+0000 7f2d62b27700 1 -- 172.21.15.91:0/4132462276 shutdown_connections 2023-11-24T06:08:07.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.540+0000 7f9ef8f26700 1 -- 172.21.15.91:0/1258664499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc005210 msgr2=0x7f9edc00dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.540+0000 7f9ef8f26700 1 --2- 172.21.15.91:0/1258664499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc005210 0x7f9edc00dc20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9eec014a00 tx=0x7f9edc005880 comp rx=0 tx=0).stop 2023-11-24T06:08:07.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.540+0000 7f8d7b7fe700 1 -- 172.21.15.91:0/3175778384 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68001670 con 0x7f8d7c06f0c0 2023-11-24T06:08:07.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.540+0000 7f8d7b7fe700 1 -- 172.21.15.91:0/3175778384 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d70004030 con 0x7f8d7c06f0c0 2023-11-24T06:08:07.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.540+0000 7f8d7b7fe700 1 -- 172.21.15.91:0/3175778384 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d700043c0 con 0x7f8d7c06f0c0 2023-11-24T06:08:07.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.540+0000 7f2d62b27700 1 -- 172.21.15.91:0/4132462276 wait complete. 2023-11-24T06:08:07.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.540+0000 7f9ef8f26700 1 -- 172.21.15.91:0/1258664499 shutdown_connections 2023-11-24T06:08:07.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.540+0000 7f9ef8f26700 1 --2- 172.21.15.91:0/1258664499 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9edc005210 0x7f9edc00dc20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.540+0000 7f9ef8f26700 1 -- 172.21.15.91:0/1258664499 >> 172.21.15.91:0/1258664499 conn(0x7f9ef40cf980 msgr2=0x7f9ef40cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.541+0000 7f8d82f04700 1 -- 172.21.15.91:0/3175778384 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d7c06f0c0 msgr2=0x7f8d7c070e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.541+0000 7f8d82f04700 1 --2- 172.21.15.91:0/3175778384 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d7c06f0c0 0x7f8d7c070e50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009840 tx=0x7f8d70009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:07.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.541+0000 7fe2fefeb700 1 -- 172.21.15.91:0/3821111437 shutdown_connections 2023-11-24T06:08:07.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.541+0000 7fe2fefeb700 1 --2- 172.21.15.91:0/3821111437 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe2f8166430 0x7fe2f80cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.541+0000 7fe2fefeb700 1 --2- 172.21.15.91:0/3821111437 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2f8145870 0x7fe2f8165ef0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.541+0000 7fe2fefeb700 1 -- 172.21.15.91:0/3821111437 >> 172.21.15.91:0/3821111437 conn(0x7fe2f80cf980 msgr2=0x7fe2f80cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.541+0000 7f8d82f04700 1 -- 172.21.15.91:0/3175778384 shutdown_connections 2023-11-24T06:08:07.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.541+0000 7f8d82f04700 1 --2- 172.21.15.91:0/3175778384 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d7c071390 0x7f8d7c0cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.541+0000 7f8d82f04700 1 --2- 172.21.15.91:0/3175778384 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d7c06f0c0 0x7f8d7c070e50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.541+0000 7f8d82f04700 1 -- 172.21.15.91:0/3175778384 >> 172.21.15.91:0/3175778384 conn(0x7f8d7c0cf980 msgr2=0x7f8d7c0cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.541+0000 7f8d82f04700 1 -- 172.21.15.91:0/3175778384 shutdown_connections 2023-11-24T06:08:07.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.542+0000 7f8d82f04700 1 -- 172.21.15.91:0/3175778384 wait complete. 2023-11-24T06:08:07.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.542+0000 7f2d62b27700 1 Processor -- start 2023-11-24T06:08:07.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.542+0000 7fe2fefeb700 1 -- 172.21.15.91:0/3821111437 shutdown_connections 2023-11-24T06:08:07.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.542+0000 7fe2fefeb700 1 -- 172.21.15.91:0/3821111437 wait complete. 2023-11-24T06:08:07.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.542+0000 7f2d62b27700 1 -- start start 2023-11-24T06:08:07.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.543+0000 7f8d82f04700 1 Processor -- start 2023-11-24T06:08:07.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.543+0000 7f2d62b27700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2d5c0d04a0 0x7f2d5c13bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.543+0000 7f2d62b27700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d5c15b170 0x7f2d5c13c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.543+0000 7f2d62b27700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2d5c13b530 0x7f2d5c200100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.543+0000 7f2d62b27700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c13ac90 con 0x7f2d5c13b530 2023-11-24T06:08:07.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.543+0000 7f2d62b27700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c15b880 con 0x7f2d5c15b170 2023-11-24T06:08:07.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.543+0000 7f2d62b27700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c15bb60 con 0x7f2d5c0d04a0 2023-11-24T06:08:07.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.543+0000 7f2d5bfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d5c15b170 0x7f2d5c13c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.543+0000 7f2d5bfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d5c15b170 0x7f2d5c13c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:42244/0 (socket says 172.21.15.91:42244) 2023-11-24T06:08:07.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.543+0000 7f2d5bfff700 1 -- 172.21.15.91:0/2585874796 learned_addr learned my addr 172.21.15.91:0/2585874796 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.543+0000 7f2d608c3700 1 --2- 172.21.15.91:0/2585874796 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2d5c0d04a0 0x7f2d5c13bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.543+0000 7fe2fefeb700 1 Processor -- start 2023-11-24T06:08:07.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f2d610c4700 1 --2- 172.21.15.91:0/2585874796 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2d5c13b530 0x7f2d5c200100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7fe2fefeb700 1 -- start start 2023-11-24T06:08:07.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f2d5bfff700 1 -- 172.21.15.91:0/2585874796 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2d5c0d04a0 msgr2=0x7f2d5c13bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f2d5bfff700 1 --2- 172.21.15.91:0/2585874796 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2d5c0d04a0 0x7f2d5c13bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f2d5bfff700 1 -- 172.21.15.91:0/2585874796 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2d5c13b530 msgr2=0x7f2d5c200100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f2d5bfff700 1 --2- 172.21.15.91:0/2585874796 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2d5c13b530 0x7f2d5c200100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f2d5bfff700 1 -- 172.21.15.91:0/2585874796 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4c009550 con 0x7f2d5c15b170 2023-11-24T06:08:07.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f2d5bfff700 1 --2- 172.21.15.91:0/2585874796 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d5c15b170 0x7f2d5c13c070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2d5400ba50 tx=0x7f2d5400bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f9ef8f26700 1 -- 172.21.15.91:0/1258664499 shutdown_connections 2023-11-24T06:08:07.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.543+0000 7f8d82f04700 1 -- start start 2023-11-24T06:08:07.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f8d82f04700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d7c06f0c0 0x7f8d7c12f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f8d82f04700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d7c071390 0x7f8d7c131c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f8d82f04700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d7c145700 0x7f8d7c0b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f8d82f04700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c159980 con 0x7f8d7c145700 2023-11-24T06:08:07.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f8d82f04700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c159c40 con 0x7f8d7c071390 2023-11-24T06:08:07.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f8d82f04700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c159f20 con 0x7f8d7c06f0c0 2023-11-24T06:08:07.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f8d80ca0700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d7c06f0c0 0x7f8d7c12f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f8d80ca0700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d7c06f0c0 0x7f8d7c12f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:41288/0 (socket says 172.21.15.91:41288) 2023-11-24T06:08:07.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f8d80ca0700 1 -- 172.21.15.91:0/4284397577 learned_addr learned my addr 172.21.15.91:0/4284397577 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f8d814a1700 1 --2- 172.21.15.91:0/4284397577 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d7c145700 0x7f8d7c0b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7f8d80ca0700 1 -- 172.21.15.91:0/4284397577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d7c071390 msgr2=0x7f8d7c131c00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:07.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7f8d80ca0700 1 --2- 172.21.15.91:0/4284397577 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d7c071390 0x7f8d7c131c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7f8d80ca0700 1 -- 172.21.15.91:0/4284397577 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d7c145700 msgr2=0x7f8d7c0b0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7f8d80ca0700 1 --2- 172.21.15.91:0/4284397577 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d7c145700 0x7f8d7c0b0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7f8d80ca0700 1 -- 172.21.15.91:0/4284397577 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d700094a0 con 0x7f8d7c06f0c0 2023-11-24T06:08:07.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7f8d80ca0700 1 --2- 172.21.15.91:0/4284397577 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d7c06f0c0 0x7f8d7c12f900 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8d6800c750 tx=0x7f8d6800cb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f2d597fa700 1 -- 172.21.15.91:0/2585874796 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5400c930 con 0x7f2d5c15b170 2023-11-24T06:08:07.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f2d597fa700 1 -- 172.21.15.91:0/2585874796 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5400cab0 con 0x7f2d5c15b170 2023-11-24T06:08:07.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7f2d597fa700 1 -- 172.21.15.91:0/2585874796 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5400ce60 con 0x7f2d5c15b170 2023-11-24T06:08:07.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.544+0000 7fde8f2b4700 1 Processor -- start 2023-11-24T06:08:07.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7fe2fefeb700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2f8070a80 0x7fe2f815e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7fe2fefeb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe2f8145870 0x7fe2f806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7fe2fefeb700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe2f8166430 0x7fe2f8137410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7fe2fefeb700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f815e590 con 0x7fe2f8166430 2023-11-24T06:08:07.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7fe2fefeb700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f806d9e0 con 0x7fe2f8070a80 2023-11-24T06:08:07.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7fe2fefeb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f806e3e0 con 0x7fe2f8145870 2023-11-24T06:08:07.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7f27a62e3700 1 Processor -- start 2023-11-24T06:08:07.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7f9ef8f26700 1 -- 172.21.15.91:0/1258664499 wait complete. 2023-11-24T06:08:07.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7fde8f2b4700 1 -- start start 2023-11-24T06:08:07.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7fde8f2b4700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fde8815db10 0x7fde8815def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7fde8f2b4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fde88071390 0x7fde8815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7fde8f2b4700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde880ce000 con 0x7fde8815a520 2023-11-24T06:08:07.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7fde8f2b4700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde880cb2d0 con 0x7fde8815db10 2023-11-24T06:08:07.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7fde8f2b4700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde880ce2c0 con 0x7fde88071390 2023-11-24T06:08:07.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7fde8d851700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fde8815a520 0x7fde8815a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.91:54594/0 (socket says 172.21.15.91:54594) 2023-11-24T06:08:07.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7fde8d851700 1 -- 172.21.15.91:0/1793402563 learned_addr learned my addr 172.21.15.91:0/1793402563 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:07.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7f27a62e3700 1 -- start start 2023-11-24T06:08:07.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7fe2fd588700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe2f8166430 0x7fe2f8137410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7fe2fd588700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe2f8166430 0x7fe2f8137410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57866/0 (socket says 172.21.15.91:57866) 2023-11-24T06:08:07.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.545+0000 7fe2fd588700 1 -- 172.21.15.91:0/2498450298 learned_addr learned my addr 172.21.15.91:0/2498450298 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7fe2fd588700 1 -- 172.21.15.91:0/2498450298 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe2f8145870 msgr2=0x7fe2f806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:07.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7fe2fd588700 1 --2- 172.21.15.91:0/2498450298 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe2f8145870 0x7fe2f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7fe2fd588700 1 -- 172.21.15.91:0/2498450298 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2f8070a80 msgr2=0x7fe2f815e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7fe2fd588700 1 --2- 172.21.15.91:0/2498450298 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe2f8070a80 0x7fe2f815e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7fe2fd588700 1 -- 172.21.15.91:0/2498450298 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e8009550 con 0x7fe2f8166430 2023-11-24T06:08:07.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7fe2fd588700 1 --2- 172.21.15.91:0/2498450298 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe2f8166430 0x7fe2f8137410 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe2f000b970 tx=0x7fe2f000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7f27a62e3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f27a014b350 0x7f27a00b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7f27a62e3700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f27a014b730 0x7f27a01661b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7f27a62e3700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f27a014c150 0x7f27a01666f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7fde8c84f700 1 --2- 172.21.15.91:0/1793402563 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fde88071390 0x7fde8815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7f27a62e3700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a00cdb30 con 0x7f27a014b730 2023-11-24T06:08:07.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7f27a62e3700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a0166c30 con 0x7f27a014c150 2023-11-24T06:08:07.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7f27a62e3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a00cd520 con 0x7f27a014b350 2023-11-24T06:08:07.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7f279ffff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f27a014b350 0x7f27a00b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7f27a4880700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f27a014c150 0x7f27a01666f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7f279ffff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f27a014b350 0x7f27a00b49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:41306/0 (socket says 172.21.15.91:41306) 2023-11-24T06:08:07.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7f27a4880700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f27a014c150 0x7f27a01666f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:42268/0 (socket says 172.21.15.91:42268) 2023-11-24T06:08:07.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7f279ffff700 1 -- 172.21.15.91:0/1261358162 learned_addr learned my addr 172.21.15.91:0/1261358162 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.572 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7f9ef8f26700 1 Processor -- start 2023-11-24T06:08:07.572 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7f279ffff700 1 -- 172.21.15.91:0/1261358162 >> v2:172.21.15.94:3300/0 conn(0x7f27a014c150 msgr2=0x7f27a01666f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.572 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7f279ffff700 1 --2- 172.21.15.91:0/1261358162 >> v2:172.21.15.94:3300/0 conn(0x7f27a014c150 0x7f27a01666f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.572 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7f279ffff700 1 -- 172.21.15.91:0/1261358162 >> v2:172.21.15.91:3300/0 conn(0x7f27a014b730 msgr2=0x7f27a01661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.572 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7f279ffff700 1 --2- 172.21.15.91:0/1261358162 >> v2:172.21.15.91:3300/0 conn(0x7f27a014b730 0x7f27a01661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.573 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7f279ffff700 1 -- 172.21.15.91:0/1261358162 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a014bf50 con 0x7f27a014b350 2023-11-24T06:08:07.573 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7f27a4880700 1 --2- 172.21.15.91:0/1261358162 >> v2:172.21.15.94:3300/0 conn(0x7f27a014c150 0x7f27a01666f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:08:07.573 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7fde8c84f700 1 -- 172.21.15.91:0/1793402563 >> v2:172.21.15.91:3300/0 conn(0x7fde8815db10 msgr2=0x7fde8815def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.573 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7fde8c84f700 1 --2- 172.21.15.91:0/1793402563 >> v2:172.21.15.91:3300/0 conn(0x7fde8815db10 0x7fde8815def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.573 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7fde8c84f700 1 -- 172.21.15.91:0/1793402563 >> v1:172.21.15.94:6789/0 conn(0x7fde8815a520 legacy=0x7fde8815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.573 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7fde8c84f700 1 -- 172.21.15.91:0/1793402563 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde880ce5a0 con 0x7fde88071390 2023-11-24T06:08:07.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7fde8c84f700 1 --2- 172.21.15.91:0/1793402563 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fde88071390 0x7fde8815e430 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fde7800a9e0 tx=0x7fde7800acf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9028ce810d7a13b4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7f279ffff700 1 --2- 172.21.15.91:0/1261358162 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f27a014b350 0x7f27a00b49e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f27a00b5690 tx=0x7f2794009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6f0851795a4dbd8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7fde7ffff700 1 -- 172.21.15.91:0/1793402563 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7800cb70 con 0x7fde88071390 2023-11-24T06:08:07.575 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7f9ef8f26700 1 -- start start 2023-11-24T06:08:07.575 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.548+0000 7f2d62b27700 1 -- 172.21.15.91:0/2585874796 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5c15be40 con 0x7f2d5c15b170 2023-11-24T06:08:07.575 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7f8d797fa700 1 -- 172.21.15.91:0/4284397577 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68004c80 con 0x7f8d7c06f0c0 2023-11-24T06:08:07.575 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7f8d797fa700 1 -- 172.21.15.91:0/4284397577 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d68004de0 con 0x7f8d7c06f0c0 2023-11-24T06:08:07.575 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7f8d797fa700 1 -- 172.21.15.91:0/4284397577 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68007310 con 0x7f8d7c06f0c0 2023-11-24T06:08:07.575 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7f8d82f04700 1 -- 172.21.15.91:0/4284397577 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7c15a200 con 0x7f8d7c06f0c0 2023-11-24T06:08:07.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.546+0000 7f8d82f04700 1 -- 172.21.15.91:0/4284397577 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7c132140 con 0x7f8d7c06f0c0 2023-11-24T06:08:07.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.548+0000 7f2d62b27700 1 -- 172.21.15.91:0/2585874796 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5c13a7a0 con 0x7f2d5c15b170 2023-11-24T06:08:07.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.547+0000 7fde7ffff700 1 -- 172.21.15.91:0/1793402563 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7800ccd0 con 0x7fde88071390 2023-11-24T06:08:07.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.548+0000 7fde7ffff700 1 -- 172.21.15.91:0/1793402563 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7800f690 con 0x7fde88071390 2023-11-24T06:08:07.577 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.548+0000 7fe2ed7fa700 1 -- 172.21.15.91:0/2498450298 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f000c9b0 con 0x7fe2f8166430 2023-11-24T06:08:07.577 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.548+0000 7fe2ed7fa700 1 -- 172.21.15.91:0/2498450298 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f000cb10 con 0x7fe2f8166430 2023-11-24T06:08:07.577 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.548+0000 7fe2ed7fa700 1 -- 172.21.15.91:0/2498450298 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0014440 con 0x7fe2f8166430 2023-11-24T06:08:07.577 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.548+0000 7f9ef8f26700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9edc005210 0x7f9ef413e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.578 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.548+0000 7f9ef8f26700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9ef41ef6d0 0x7f9ef41efb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.578 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f2d62b27700 1 -- 172.21.15.91:0/2585874796 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2d5c13aa80 con 0x7f2d5c15b170 2023-11-24T06:08:07.578 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7fde8f2b4700 1 -- 172.21.15.91:0/1793402563 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fde88071390 msgr2=0x7fde8815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.578 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7fde8f2b4700 1 --2- 172.21.15.91:0/1793402563 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fde88071390 0x7fde8815e430 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fde7800a9e0 tx=0x7fde7800acf0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.579 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.548+0000 7f279effd700 1 -- 172.21.15.91:0/1261358162 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2794011830 con 0x7f27a014b350 2023-11-24T06:08:07.579 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.548+0000 7f279effd700 1 -- 172.21.15.91:0/1261358162 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2794011990 con 0x7f27a014b350 2023-11-24T06:08:07.579 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.548+0000 7f279effd700 1 -- 172.21.15.91:0/1261358162 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2794011d20 con 0x7f27a014b350 2023-11-24T06:08:07.579 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f27a62e3700 1 -- 172.21.15.91:0/1261358162 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f27a014b350 msgr2=0x7f27a00b49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.579 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f27a62e3700 1 --2- 172.21.15.91:0/1261358162 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f27a014b350 0x7f27a00b49e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f27a00b5690 tx=0x7f2794009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f9ef8f26700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ef41f0090 0x7f9ef41f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f9ef8f26700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef41f13a0 con 0x7f9edc005210 2023-11-24T06:08:07.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f9ef8f26700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef414da80 con 0x7f9ef41f0090 2023-11-24T06:08:07.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f9ef8f26700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef414dd60 con 0x7f9ef41ef6d0 2023-11-24T06:08:07.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f9ef1d9b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9ef41ef6d0 0x7f9ef41efb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f9ef1d9b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9ef41ef6d0 0x7f9ef41efb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:41312/0 (socket says 172.21.15.91:41312) 2023-11-24T06:08:07.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f9ef1d9b700 1 -- 172.21.15.91:0/1554500218 learned_addr learned my addr 172.21.15.91:0/1554500218 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f9ef2d9d700 1 --2- 172.21.15.91:0/1554500218 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ef41f0090 0x7f9ef41f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.581 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7fe2fefeb700 1 -- 172.21.15.91:0/2498450298 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f806e6c0 con 0x7fe2f8166430 2023-11-24T06:08:07.581 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7fe2fefeb700 1 -- 172.21.15.91:0/2498450298 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f806ec80 con 0x7fe2f8166430 2023-11-24T06:08:07.581 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f9ef259c700 1 --2- 172.21.15.91:0/1554500218 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9edc005210 0x7f9ef413e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.581 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f9ef1d9b700 1 -- 172.21.15.91:0/1554500218 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ef41f0090 msgr2=0x7f9ef41f2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.582 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f9ef1d9b700 1 --2- 172.21.15.91:0/1554500218 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9ef41f0090 0x7f9ef41f2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.582 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f9ef1d9b700 1 -- 172.21.15.91:0/1554500218 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9edc005210 msgr2=0x7f9ef413e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.582 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f9ef1d9b700 1 --2- 172.21.15.91:0/1554500218 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9edc005210 0x7f9ef413e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.582 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f9ef1d9b700 1 -- 172.21.15.91:0/1554500218 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef41f0ce0 con 0x7f9ef41ef6d0 2023-11-24T06:08:07.582 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.550+0000 7f8d797fa700 1 -- 172.21.15.91:0/4284397577 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8d68007470 con 0x7f8d7c06f0c0 2023-11-24T06:08:07.583 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f27a62e3700 1 -- 172.21.15.91:0/1261358162 shutdown_connections 2023-11-24T06:08:07.583 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f27a62e3700 1 --2- 172.21.15.91:0/1261358162 >> v2:172.21.15.94:3300/0 conn(0x7f27a014c150 0x7f27a01666f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.583 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f27a62e3700 1 --2- 172.21.15.91:0/1261358162 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f27a014b350 0x7f27a00b49e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.583 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f27a62e3700 1 --2- 172.21.15.91:0/1261358162 >> v2:172.21.15.91:3300/0 conn(0x7f27a014b730 0x7f27a01661b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.583 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.549+0000 7f27a62e3700 1 -- 172.21.15.91:0/1261358162 >> 172.21.15.91:0/1261358162 conn(0x7f27a00b1cc0 msgr2=0x7f27a00b20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.550+0000 7f8d797fa700 1 --2- 172.21.15.91:0/4284397577 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8d6c06b450 0x7f8d6c06d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.550+0000 7f8d797fa700 1 -- 172.21.15.91:0/4284397577 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d68088ba0 con 0x7f8d7c06f0c0 2023-11-24T06:08:07.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.550+0000 7f8d7bfff700 1 --2- 172.21.15.91:0/4284397577 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8d6c06b450 0x7f8d6c06d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.550+0000 7f9ef1d9b700 1 --2- 172.21.15.91:0/1554500218 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9ef41ef6d0 0x7f9ef41efb50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9ee400f860 tx=0x7f9ee400fb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.550+0000 7fde8f2b4700 1 -- 172.21.15.91:0/1793402563 shutdown_connections 2023-11-24T06:08:07.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.551+0000 7fde8f2b4700 1 --2- 172.21.15.91:0/1793402563 >> v2:172.21.15.91:3300/0 conn(0x7fde8815db10 0x7fde8815def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.551+0000 7fde8f2b4700 1 --2- 172.21.15.91:0/1793402563 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fde88071390 0x7fde8815e430 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.551+0000 7fde8f2b4700 1 -- 172.21.15.91:0/1793402563 >> 172.21.15.91:0/1793402563 conn(0x7fde880cf980 msgr2=0x7fde880cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.551+0000 7f2d597fa700 1 -- 172.21.15.91:0/2585874796 <== mon.1 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2d5401d3b0 con 0x7f2d5c15b170 2023-11-24T06:08:07.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.551+0000 7f9ee2ffd700 1 -- 172.21.15.91:0/1554500218 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4009040 con 0x7f9ef41ef6d0 2023-11-24T06:08:07.586 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.551+0000 7f9ee2ffd700 1 -- 172.21.15.91:0/1554500218 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee400d9b0 con 0x7f9ef41ef6d0 2023-11-24T06:08:07.586 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.551+0000 7f8d7bfff700 1 --2- 172.21.15.91:0/4284397577 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8d6c06b450 0x7f8d6c06d910 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009b90 tx=0x7f8d70008000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.586 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.551+0000 7f2d597fa700 1 --2- 172.21.15.91:0/2585874796 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2d4006af20 0x7f2d4006d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.586 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.551+0000 7f2d597fa700 1 -- 172.21.15.91:0/2585874796 <== mon.1 v2:172.21.15.94:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d54006e40 con 0x7f2d5c15b170 2023-11-24T06:08:07.587 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.551+0000 7fe2ed7fa700 1 -- 172.21.15.91:0/2498450298 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe2f00145a0 con 0x7fe2f8166430 2023-11-24T06:08:07.587 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.551+0000 7f2d597fa700 1 --2- 172.21.15.91:0/2585874796 >> [v2:172.21.15.94:6824/3888015416,v1:172.21.15.94:6825/3888015416] conn(0x7f2d40073140 0x7f2d40075560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.587 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.551+0000 7f2d597fa700 1 -- 172.21.15.91:0/2585874796 --> [v2:172.21.15.94:6824/3888015416,v1:172.21.15.94:6825/3888015416] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2d40075c10 con 0x7f2d40073140 2023-11-24T06:08:07.587 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.551+0000 7f2d597fa700 1 -- 172.21.15.91:0/2585874796 <== mon.1 v2:172.21.15.94:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d5408b610 con 0x7f2d5c15b170 2023-11-24T06:08:07.588 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.552+0000 7f2d608c3700 1 --2- 172.21.15.91:0/2585874796 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2d4006af20 0x7f2d4006d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.589 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.551+0000 7fde8f2b4700 1 -- 172.21.15.91:0/1793402563 shutdown_connections 2023-11-24T06:08:07.589 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.551+0000 7f9ee2ffd700 1 -- 172.21.15.91:0/1554500218 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee400dd60 con 0x7f9ef41ef6d0 2023-11-24T06:08:07.589 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.552+0000 7fe2ed7fa700 1 --2- 172.21.15.91:0/2498450298 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe2e406af20 0x7fe2e406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.590 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.552+0000 7fe2ed7fa700 1 -- 172.21.15.91:0/2498450298 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2f0089110 con 0x7fe2f8166430 2023-11-24T06:08:07.590 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.552+0000 7fde8f2b4700 1 -- 172.21.15.91:0/1793402563 wait complete. 2023-11-24T06:08:07.590 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.552+0000 7f8d82f04700 1 --2- 172.21.15.91:0/4284397577 >> [v2:172.21.15.91:6808/649116497,v1:172.21.15.91:6809/649116497] conn(0x7f8d5c0015d0 0x7f8d5c003a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.590 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.552+0000 7f8d82f04700 1 -- 172.21.15.91:0/4284397577 --> [v2:172.21.15.91:6808/649116497,v1:172.21.15.91:6809/649116497] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d5c006bc0 con 0x7f8d5c0015d0 2023-11-24T06:08:07.591 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.552+0000 7f2d608c3700 1 --2- 172.21.15.91:0/2585874796 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2d4006af20 0x7f2d4006d3e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2d50006fd0 tx=0x7f2d50009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.591 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.552+0000 7fe2fcd87700 1 --2- 172.21.15.91:0/2498450298 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe2e406af20 0x7fe2e406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.591 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.552+0000 7f2d610c4700 1 --2- 172.21.15.91:0/2585874796 >> [v2:172.21.15.94:6824/3888015416,v1:172.21.15.94:6825/3888015416] conn(0x7f2d40073140 0x7f2d40075560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.591 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.552+0000 7f8d814a1700 1 --2- 172.21.15.91:0/4284397577 >> [v2:172.21.15.91:6808/649116497,v1:172.21.15.91:6809/649116497] conn(0x7f8d5c0015d0 0x7f8d5c003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.592 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.552+0000 7ff89d945700 1 Processor -- start 2023-11-24T06:08:07.592 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.553+0000 7ff89d945700 1 -- start start 2023-11-24T06:08:07.592 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.553+0000 7fe2fefeb700 1 --2- 172.21.15.91:0/2498450298 >> [v2:172.21.15.91:6816/420345853,v1:172.21.15.91:6819/420345853] conn(0x7fe2f8132d10 0x7fe2f8137950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.592 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.553+0000 7fe2fefeb700 1 -- 172.21.15.91:0/2498450298 --> [v2:172.21.15.91:6816/420345853,v1:172.21.15.91:6819/420345853] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe2f8067f60 con 0x7fe2f8132d10 2023-11-24T06:08:07.593 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.553+0000 7f8d814a1700 1 --2- 172.21.15.91:0/4284397577 >> [v2:172.21.15.91:6808/649116497,v1:172.21.15.91:6809/649116497] conn(0x7f8d5c0015d0 0x7f8d5c003a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.593 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.553+0000 7f27a62e3700 1 -- 172.21.15.91:0/1261358162 shutdown_connections 2023-11-24T06:08:07.593 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.553+0000 7f27a62e3700 1 -- 172.21.15.91:0/1261358162 wait complete. 2023-11-24T06:08:07.593 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.553+0000 7fe2effff700 1 --2- 172.21.15.91:0/2498450298 >> [v2:172.21.15.91:6816/420345853,v1:172.21.15.91:6819/420345853] conn(0x7fe2f8132d10 0x7fe2f8137950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.594 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.553+0000 7fe2effff700 1 --2- 172.21.15.91:0/2498450298 >> [v2:172.21.15.91:6816/420345853,v1:172.21.15.91:6819/420345853] conn(0x7fe2f8132d10 0x7fe2f8137950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.594 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7fde8f2b4700 1 Processor -- start 2023-11-24T06:08:07.594 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7ff89d945700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff8980cf800 0x7ff8980b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.594 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7ff89d945700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff8981448c0 0x7ff8980b4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.594 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7ff89d945700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff898071340 con 0x7ff898070eb0 2023-11-24T06:08:07.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7ff89d945700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8980b3060 con 0x7ff8980cf800 2023-11-24T06:08:07.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7ff89d945700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8980b3340 con 0x7ff8981448c0 2023-11-24T06:08:07.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7ff8967fc700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff8980cf800 0x7ff8980b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7ff8967fc700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff8980cf800 0x7ff8980b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57898/0 (socket says 172.21.15.91:57898) 2023-11-24T06:08:07.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7ff8967fc700 1 -- 172.21.15.91:0/4273766467 learned_addr learned my addr 172.21.15.91:0/4273766467 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7fde8f2b4700 1 -- start start 2023-11-24T06:08:07.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7fe2fcd87700 1 --2- 172.21.15.91:0/2498450298 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe2e406af20 0x7fe2e406d3e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8136190 tx=0x7fe2f4007a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7ff8977fe700 1 --2- 172.21.15.91:0/4273766467 >> v2:172.21.15.94:3300/0 conn(0x7ff8981448c0 0x7ff8980b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7ff8967fc700 1 -- 172.21.15.91:0/4273766467 >> v2:172.21.15.94:3300/0 conn(0x7ff8981448c0 msgr2=0x7ff8980b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7ff8967fc700 1 --2- 172.21.15.91:0/4273766467 >> v2:172.21.15.94:3300/0 conn(0x7ff8981448c0 0x7ff8980b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7ff8967fc700 1 -- 172.21.15.91:0/4273766467 >> v1:172.21.15.94:6789/0 conn(0x7ff898070eb0 legacy=0x7ff898144110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7ff8967fc700 1 -- 172.21.15.91:0/4273766467 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8980b3620 con 0x7ff8980cf800 2023-11-24T06:08:07.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.555+0000 7f2d610c4700 1 --2- 172.21.15.91:0/2585874796 >> [v2:172.21.15.94:6824/3888015416,v1:172.21.15.94:6825/3888015416] conn(0x7f2d40073140 0x7f2d40075560 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.555+0000 7ff8967fc700 1 --2- 172.21.15.91:0/4273766467 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8980cf800 0x7ff8980b3e20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff8980cc170 tx=0x7ff88c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edc2fe9582c205e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.555+0000 7ff895ffb700 1 -- 172.21.15.91:0/4273766467 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c011830 con 0x7ff8980cf800 2023-11-24T06:08:07.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.555+0000 7ff895ffb700 1 -- 172.21.15.91:0/4273766467 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88c011990 con 0x7ff8980cf800 2023-11-24T06:08:07.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.555+0000 7ff895ffb700 1 -- 172.21.15.91:0/4273766467 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c011d20 con 0x7ff8980cf800 2023-11-24T06:08:07.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7f27a62e3700 1 Processor -- start 2023-11-24T06:08:07.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.555+0000 7f27a62e3700 1 -- start start 2023-11-24T06:08:07.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.555+0000 7fde8f2b4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fde88071390 0x7fde8813abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.555+0000 7fde8f2b4700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fde8815a520 0x7fde880b5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.555+0000 7fde8f2b4700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde8815db10 0x7fde881f3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.555+0000 7fde8f2b4700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8813beb0 con 0x7fde8815a520 2023-11-24T06:08:07.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.555+0000 7fde8f2b4700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde880b58c0 con 0x7fde8815db10 2023-11-24T06:08:07.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.555+0000 7fde8f2b4700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde88138ae0 con 0x7fde88071390 2023-11-24T06:08:07.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.555+0000 7fde8d050700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fde88071390 0x7fde8813abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.555+0000 7fde8d050700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fde88071390 0x7fde8813abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:41336/0 (socket says 172.21.15.91:41336) 2023-11-24T06:08:07.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.555+0000 7fde8d050700 1 -- 172.21.15.91:0/4277528224 learned_addr learned my addr 172.21.15.91:0/4277528224 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.600 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.555+0000 7fde8c84f700 1 --2- 172.21.15.91:0/4277528224 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fde8815a520 0x7fde880b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.600 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7f9ef8f26700 1 -- 172.21.15.91:0/1554500218 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef41f0fc0 con 0x7f9ef41ef6d0 2023-11-24T06:08:07.600 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7f27a62e3700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27a014b350 0x7f27a00ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.600 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7f27a62e3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f27a014b730 0x7f27a00abd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.600 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7f27a62e3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27a014c150 0x7f27a0164780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7f27a62e3700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a01370f0 con 0x7f27a014b350 2023-11-24T06:08:07.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7f27a62e3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a01642f0 con 0x7f27a014c150 2023-11-24T06:08:07.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7f27a62e3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a00aeef0 con 0x7f27a014b730 2023-11-24T06:08:07.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7f9ef8f26700 1 -- 172.21.15.91:0/1554500218 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef414d6a0 con 0x7f9ef41ef6d0 2023-11-24T06:08:07.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7fde8c84f700 1 -- 172.21.15.91:0/4277528224 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fde88071390 msgr2=0x7fde8813abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7fde8c84f700 1 --2- 172.21.15.91:0/4277528224 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fde88071390 0x7fde8813abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7fde8c84f700 1 -- 172.21.15.91:0/4277528224 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde8815db10 msgr2=0x7fde881f3af0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:07.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7fde8c84f700 1 --2- 172.21.15.91:0/4277528224 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fde8815db10 0x7fde881f3af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7f2d597fa700 1 -- 172.21.15.91:0/2585874796 <== osd.4 v2:172.21.15.94:6824/3888015416 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f2d40075c10 con 0x7f2d40073140 2023-11-24T06:08:07.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7fde8c84f700 1 -- 172.21.15.91:0/4277528224 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7800a640 con 0x7fde8815a520 2023-11-24T06:08:07.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7fde8c84f700 1 --2- 172.21.15.91:0/4277528224 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fde8815a520 0x7fde880b5380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde78018000 tx=0x7fde78004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7fde7dffb700 1 -- 172.21.15.91:0/4277528224 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78004560 con 0x7fde8815a520 2023-11-24T06:08:07.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7f279ffff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27a014b350 0x7f27a00ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7f279ffff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27a014b350 0x7f27a00ae5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57908/0 (socket says 172.21.15.91:57908) 2023-11-24T06:08:07.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7f279ffff700 1 -- 172.21.15.91:0/1767100752 learned_addr learned my addr 172.21.15.91:0/1767100752 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7fe2ed7fa700 1 -- 172.21.15.91:0/2498450298 <== osd.0 v2:172.21.15.91:6816/420345853 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fe2f8067f60 con 0x7fe2f8132d10 2023-11-24T06:08:07.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.554+0000 7f8d797fa700 1 -- 172.21.15.91:0/4284397577 <== osd.2 v2:172.21.15.91:6808/649116497 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f8d5c006bc0 con 0x7f8d5c0015d0 2023-11-24T06:08:07.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7f9ef8f26700 1 -- 172.21.15.91:0/1554500218 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9ef4157da0 con 0x7f9ef41ef6d0 2023-11-24T06:08:07.605 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7ff89d945700 1 -- 172.21.15.91:0/4273766467 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8980cf800 msgr2=0x7ff8980b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.605 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7ff89d945700 1 --2- 172.21.15.91:0/4273766467 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8980cf800 0x7ff8980b3e20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff8980cc170 tx=0x7ff88c009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.605 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7f27a4880700 1 --2- 172.21.15.91:0/1767100752 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27a014c150 0x7f27a0164780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.605 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7f279f7fe700 1 --2- 172.21.15.91:0/1767100752 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f27a014b730 0x7f27a00abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7f279ffff700 1 -- 172.21.15.91:0/1767100752 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f27a014b730 msgr2=0x7f27a00abd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7f279ffff700 1 --2- 172.21.15.91:0/1767100752 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f27a014b730 0x7f27a00abd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7f279ffff700 1 -- 172.21.15.91:0/1767100752 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27a014c150 msgr2=0x7f27a0164780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7f279ffff700 1 --2- 172.21.15.91:0/1767100752 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27a014c150 0x7f27a0164780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7f279ffff700 1 -- 172.21.15.91:0/1767100752 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2798009a70 con 0x7f27a014b350 2023-11-24T06:08:07.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7f27a4880700 1 --2- 172.21.15.91:0/1767100752 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27a014c150 0x7f27a0164780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:07.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7f279ffff700 1 --2- 172.21.15.91:0/1767100752 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27a014b350 0x7f27a00ae5a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2794000c00 tx=0x7f2794004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7ff89d945700 1 -- 172.21.15.91:0/4273766467 shutdown_connections 2023-11-24T06:08:07.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7ff89d945700 1 --2- 172.21.15.91:0/4273766467 >> v2:172.21.15.94:3300/0 conn(0x7ff8981448c0 0x7ff8980b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7ff89d945700 1 --2- 172.21.15.91:0/4273766467 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8980cf800 0x7ff8980b3e20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7ff89d945700 1 -- 172.21.15.91:0/4273766467 >> 172.21.15.91:0/4273766467 conn(0x7ff8980b1cc0 msgr2=0x7ff8980b20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7f279cff9700 1 -- 172.21.15.91:0/1767100752 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2794004510 con 0x7f27a014b350 2023-11-24T06:08:07.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7f279cff9700 1 -- 172.21.15.91:0/1767100752 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27940036a0 con 0x7f27a014b350 2023-11-24T06:08:07.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7f279cff9700 1 -- 172.21.15.91:0/1767100752 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2794003a50 con 0x7f27a014b350 2023-11-24T06:08:07.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.558+0000 7f27a62e3700 1 -- 172.21.15.91:0/1767100752 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2794009550 con 0x7f27a014b350 2023-11-24T06:08:07.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.558+0000 7f27a62e3700 1 -- 172.21.15.91:0/1767100752 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a0163ca0 con 0x7f27a014b350 2023-11-24T06:08:07.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7fde7dffb700 1 -- 172.21.15.91:0/4277528224 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7800fa50 con 0x7fde8815a520 2023-11-24T06:08:07.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7fde7dffb700 1 -- 172.21.15.91:0/4277528224 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7800fdc0 con 0x7fde8815a520 2023-11-24T06:08:07.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.556+0000 7fde8f2b4700 1 -- 172.21.15.91:0/4277528224 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde881f4030 con 0x7fde8815a520 2023-11-24T06:08:07.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.557+0000 7fde8f2b4700 1 -- 172.21.15.91:0/4277528224 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8813b510 con 0x7fde8815a520 2023-11-24T06:08:07.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.559+0000 7fde8f2b4700 1 -- 172.21.15.91:0/4277528224 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fde68000fb0 con 0x7fde8815a520 2023-11-24T06:08:07.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.559+0000 7f27a62e3700 1 -- 172.21.15.91:0/1767100752 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2780000fb0 con 0x7f27a014b350 2023-11-24T06:08:07.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.559+0000 7f9ee2ffd700 1 -- 172.21.15.91:0/1554500218 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9ee4012440 con 0x7f9ef41ef6d0 2023-11-24T06:08:07.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.559+0000 7f9ee2ffd700 1 --2- 172.21.15.91:0/1554500218 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9ed406aed0 0x7f9ed406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.559+0000 7f9ee2ffd700 1 -- 172.21.15.91:0/1554500218 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ee4089e30 con 0x7f9ef41ef6d0 2023-11-24T06:08:07.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.560+0000 7f9ee2ffd700 1 --2- 172.21.15.91:0/1554500218 >> [v2:172.21.15.91:6813/66406286,v1:172.21.15.91:6815/66406286] conn(0x7f9ed4073060 0x7f9ed4075480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.560+0000 7f9ee2ffd700 1 -- 172.21.15.91:0/1554500218 --> [v2:172.21.15.91:6813/66406286,v1:172.21.15.91:6815/66406286] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ed4075b30 con 0x7f9ed4073060 2023-11-24T06:08:07.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.560+0000 7f9ee2ffd700 1 -- 172.21.15.91:0/1554500218 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ee408c450 con 0x7f9ef41ef6d0 2023-11-24T06:08:07.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.560+0000 7f9ef2d9d700 1 --2- 172.21.15.91:0/1554500218 >> [v2:172.21.15.91:6813/66406286,v1:172.21.15.91:6815/66406286] conn(0x7f9ed4073060 0x7f9ed4075480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.561+0000 7ff89d945700 1 -- 172.21.15.91:0/4273766467 shutdown_connections 2023-11-24T06:08:07.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.559+0000 7f7f0e1a7700 1 Processor -- start 2023-11-24T06:08:07.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.559+0000 7f7f0e1a7700 1 -- start start 2023-11-24T06:08:07.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.560+0000 7f7f0e1a7700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f7f0806cae0 0x7f7f080ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.560+0000 7f7f0e1a7700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7f080ce980 0x7f7f080cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.560+0000 7f7f0e1a7700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f080cab10 con 0x7f7f08070a80 2023-11-24T06:08:07.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.560+0000 7f7f0e1a7700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f080c9fd0 con 0x7f7f0806cae0 2023-11-24T06:08:07.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.560+0000 7f7f0e1a7700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f080cd7e0 con 0x7f7f080ce980 2023-11-24T06:08:07.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.561+0000 7ff89d945700 1 -- 172.21.15.91:0/4273766467 wait complete. 2023-11-24T06:08:07.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.560+0000 7f9ef2d9d700 1 --2- 172.21.15.91:0/1554500218 >> [v2:172.21.15.91:6813/66406286,v1:172.21.15.91:6815/66406286] conn(0x7f9ed4073060 0x7f9ed4075480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.561+0000 7f9ef259c700 1 --2- 172.21.15.91:0/1554500218 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9ed406aed0 0x7f9ed406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.560+0000 7f279cff9700 1 -- 172.21.15.91:0/1767100752 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2794003bb0 con 0x7f27a014b350 2023-11-24T06:08:07.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.560+0000 7f279cff9700 1 --2- 172.21.15.91:0/1767100752 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f278806b080 0x7f278806d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.560+0000 7f279cff9700 1 -- 172.21.15.91:0/1767100752 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f279408ab30 con 0x7f27a014b350 2023-11-24T06:08:07.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.560+0000 7f279cff9700 1 --2- 172.21.15.91:0/1767100752 >> [v2:172.21.15.91:6800/1071490516,v1:172.21.15.91:6801/1071490516] conn(0x7f2788073220 0x7f2788075640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.560+0000 7f279cff9700 1 -- 172.21.15.91:0/1767100752 --> [v2:172.21.15.91:6800/1071490516,v1:172.21.15.91:6801/1071490516] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2788075cf0 con 0x7f2788073220 2023-11-24T06:08:07.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.561+0000 7f27a4880700 1 --2- 172.21.15.91:0/1767100752 >> [v2:172.21.15.91:6800/1071490516,v1:172.21.15.91:6801/1071490516] conn(0x7f2788073220 0x7f2788075640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.561+0000 7f279f7fe700 1 --2- 172.21.15.91:0/1767100752 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f278806b080 0x7f278806d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.561+0000 7f27a4880700 1 --2- 172.21.15.91:0/1767100752 >> [v2:172.21.15.91:6800/1071490516,v1:172.21.15.91:6801/1071490516] conn(0x7f2788073220 0x7f2788075640 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.561+0000 7f279f7fe700 1 --2- 172.21.15.91:0/1767100752 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f278806b080 0x7f278806d540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2790006fd0 tx=0x7f2790008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.561+0000 7fde7dffb700 1 -- 172.21.15.91:0/4277528224 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fde780036a0 con 0x7fde8815a520 2023-11-24T06:08:07.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.562+0000 7fde7dffb700 1 --2- 172.21.15.91:0/4277528224 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fde6c06aed0 0x7fde6c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.562+0000 7fde7dffb700 1 -- 172.21.15.91:0/4277528224 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde78013070 con 0x7fde8815a520 2023-11-24T06:08:07.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.562+0000 7fde7dffb700 1 --2- 172.21.15.91:0/4277528224 >> [v2:172.21.15.94:6800/4250977597,v1:172.21.15.94:6801/4250977597] conn(0x7fde6c073070 0x7fde6c075490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.562+0000 7fde7dffb700 1 -- 172.21.15.91:0/4277528224 --> [v2:172.21.15.94:6800/4250977597,v1:172.21.15.94:6801/4250977597] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fde6c075b40 con 0x7fde6c073070 2023-11-24T06:08:07.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.562+0000 7f279cff9700 1 -- 172.21.15.91:0/1767100752 <== osd.3 v2:172.21.15.91:6800/1071490516 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f2788075cf0 con 0x7f2788073220 2023-11-24T06:08:07.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.563+0000 7f9ef259c700 1 --2- 172.21.15.91:0/1554500218 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9ed406aed0 0x7f9ed406d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8007a70 tx=0x7f9ee8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.563+0000 7ff89d945700 1 Processor -- start 2023-11-24T06:08:07.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.563+0000 7fde8d050700 1 --2- 172.21.15.91:0/4277528224 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fde6c06aed0 0x7fde6c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.563+0000 7fde8d851700 1 --2- 172.21.15.91:0/4277528224 >> [v2:172.21.15.94:6800/4250977597,v1:172.21.15.94:6801/4250977597] conn(0x7fde6c073070 0x7fde6c075490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.563+0000 7fde8d050700 1 --2- 172.21.15.91:0/4277528224 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fde6c06aed0 0x7fde6c06d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fde84009ad0 tx=0x7fde84009290 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.563+0000 7ff89d945700 1 -- start start 2023-11-24T06:08:07.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.564+0000 7ff89d945700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8980cf800 0x7ff89816b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.564+0000 7ff89d945700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8981448c0 0x7ff8980726c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.564+0000 7ff89d945700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff89815d0f0 0x7ff898141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.564+0000 7ff89d945700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89815d580 con 0x7ff8981448c0 2023-11-24T06:08:07.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.564+0000 7ff89d945700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89815d860 con 0x7ff8980cf800 2023-11-24T06:08:07.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.564+0000 7ff89d945700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff89815db40 con 0x7ff89815d0f0 2023-11-24T06:08:07.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.564+0000 7ff8967fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8981448c0 0x7ff8980726c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.564+0000 7ff8967fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8981448c0 0x7ff8980726c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57936/0 (socket says 172.21.15.91:57936) 2023-11-24T06:08:07.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.564+0000 7ff8967fc700 1 -- 172.21.15.91:0/513294259 learned_addr learned my addr 172.21.15.91:0/513294259 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.565+0000 7ff8977fe700 1 --2- 172.21.15.91:0/513294259 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff89815d0f0 0x7ff898141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.565+0000 7ff896ffd700 1 --2- 172.21.15.91:0/513294259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8980cf800 0x7ff89816b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.617 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.565+0000 7ff8967fc700 1 -- 172.21.15.91:0/513294259 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff89815d0f0 msgr2=0x7ff898141b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.617 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.565+0000 7ff8967fc700 1 --2- 172.21.15.91:0/513294259 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff89815d0f0 0x7ff898141b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.617 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.565+0000 7ff8967fc700 1 -- 172.21.15.91:0/513294259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8980cf800 msgr2=0x7ff89816b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.617 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.565+0000 7ff8967fc700 1 --2- 172.21.15.91:0/513294259 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8980cf800 0x7ff89816b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.617 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.565+0000 7ff8967fc700 1 -- 172.21.15.91:0/513294259 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c009550 con 0x7ff8981448c0 2023-11-24T06:08:07.617 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.565+0000 7fde8d851700 1 --2- 172.21.15.91:0/4277528224 >> [v2:172.21.15.94:6800/4250977597,v1:172.21.15.94:6801/4250977597] conn(0x7fde6c073070 0x7fde6c075490 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.617 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.565+0000 7ff8967fc700 1 --2- 172.21.15.91:0/513294259 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8981448c0 0x7ff8980726c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff88c009fd0 tx=0x7ff88c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.563+0000 7f7f06ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f7f0806cae0 0x7f7f080ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.563+0000 7f7f06ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f7f0806cae0 0x7f7f080ce440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57926/0 (socket says 172.21.15.91:57926) 2023-11-24T06:08:07.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.563+0000 7f7f06ffd700 1 -- 172.21.15.91:0/2318914280 learned_addr learned my addr 172.21.15.91:0/2318914280 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.563+0000 7f7f06ffd700 1 -- 172.21.15.91:0/2318914280 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7f080ce980 msgr2=0x7f7f080cafa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T06:08:07.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.563+0000 7f7f06ffd700 1 --2- 172.21.15.91:0/2318914280 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7f080ce980 0x7f7f080cafa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.563+0000 7f7f06ffd700 1 -- 172.21.15.91:0/2318914280 >> v1:172.21.15.91:6789/0 conn(0x7f7f08070a80 legacy=0x7f7f08070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.563+0000 7f7f06ffd700 1 -- 172.21.15.91:0/2318914280 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f080ca1e0 con 0x7f7f0806cae0 2023-11-24T06:08:07.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.564+0000 7f7f06ffd700 1 --2- 172.21.15.91:0/2318914280 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f0806cae0 0x7f7f080ce440 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00aa90 tx=0x7f7efc009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28bfb7e7e2bd686 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.566+0000 7fde7dffb700 1 -- 172.21.15.91:0/4277528224 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde780563a0 con 0x7fde8815a520 2023-11-24T06:08:07.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.566+0000 7f7f067fc700 1 -- 172.21.15.91:0/2318914280 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc019c60 con 0x7f7f0806cae0 2023-11-24T06:08:07.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.566+0000 7f7f067fc700 1 -- 172.21.15.91:0/2318914280 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7efc019dc0 con 0x7f7f0806cae0 2023-11-24T06:08:07.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.566+0000 7f7f067fc700 1 -- 172.21.15.91:0/2318914280 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc009d50 con 0x7f7f0806cae0 2023-11-24T06:08:07.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.566+0000 7f279cff9700 1 -- 172.21.15.91:0/1767100752 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2794055c70 con 0x7f27a014b350 2023-11-24T06:08:07.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.567+0000 7fde7dffb700 1 -- 172.21.15.91:0/4277528224 <== osd.6 v2:172.21.15.94:6800/4250977597 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fde6c075b40 con 0x7fde6c073070 2023-11-24T06:08:07.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.567+0000 7f9ee2ffd700 1 -- 172.21.15.91:0/1554500218 <== osd.1 v2:172.21.15.91:6813/66406286 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9ed4075b30 con 0x7f9ed4073060 2023-11-24T06:08:07.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.568+0000 7ff887fff700 1 -- 172.21.15.91:0/513294259 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c0036a0 con 0x7ff8981448c0 2023-11-24T06:08:07.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.568+0000 7ff887fff700 1 -- 172.21.15.91:0/513294259 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88c01f070 con 0x7ff8981448c0 2023-11-24T06:08:07.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.568+0000 7ff887fff700 1 -- 172.21.15.91:0/513294259 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c003a60 con 0x7ff8981448c0 2023-11-24T06:08:07.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.568+0000 7ff89d945700 1 -- 172.21.15.91:0/513294259 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89815de20 con 0x7ff8981448c0 2023-11-24T06:08:07.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.568+0000 7ff89d945700 1 -- 172.21.15.91:0/513294259 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff898072c00 con 0x7ff8981448c0 2023-11-24T06:08:07.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.572+0000 7ff887fff700 1 -- 172.21.15.91:0/513294259 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff88c003bc0 con 0x7ff8981448c0 2023-11-24T06:08:07.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.572+0000 7ff887fff700 1 --2- 172.21.15.91:0/513294259 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff87c06af20 0x7ff87c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.572+0000 7ff887fff700 1 -- 172.21.15.91:0/513294259 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff88c021070 con 0x7ff8981448c0 2023-11-24T06:08:07.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.572+0000 7ff896ffd700 1 --2- 172.21.15.91:0/513294259 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff87c06af20 0x7ff87c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.573+0000 7ff896ffd700 1 --2- 172.21.15.91:0/513294259 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff87c06af20 0x7ff87c06d3e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff8980cfe50 tx=0x7ff888009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.578+0000 7fe2fefeb700 1 -- 172.21.15.91:0/2498450298 --> [v2:172.21.15.91:6816/420345853,v1:172.21.15.91:6819/420345853] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe2f80c9db0 con 0x7fe2f8132d10 2023-11-24T06:08:07.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.578+0000 7fe2ed7fa700 1 -- 172.21.15.91:0/2498450298 <== osd.0 v2:172.21.15.91:6816/420345853 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe2f80c9db0 con 0x7fe2f8132d10 2023-11-24T06:08:07.623 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.579+0000 7fe2fefeb700 1 -- 172.21.15.91:0/2498450298 >> [v2:172.21.15.91:6816/420345853,v1:172.21.15.91:6819/420345853] conn(0x7fe2f8132d10 msgr2=0x7fe2f8137950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.623 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.579+0000 7fe2fefeb700 1 --2- 172.21.15.91:0/2498450298 >> [v2:172.21.15.91:6816/420345853,v1:172.21.15.91:6819/420345853] conn(0x7fe2f8132d10 0x7fe2f8137950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.623 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.580+0000 7fe2fefeb700 1 -- 172.21.15.91:0/2498450298 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe2e406af20 msgr2=0x7fe2e406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.623 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.580+0000 7fe2fefeb700 1 --2- 172.21.15.91:0/2498450298 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe2e406af20 0x7fe2e406d3e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8136190 tx=0x7fe2f4007a20 comp rx=0 tx=0).stop 2023-11-24T06:08:07.623 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.580+0000 7fe2fefeb700 1 -- 172.21.15.91:0/2498450298 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe2f8166430 msgr2=0x7fe2f8137410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.624 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.580+0000 7fe2fefeb700 1 --2- 172.21.15.91:0/2498450298 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe2f8166430 0x7fe2f8137410 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe2f000b970 tx=0x7fe2f000bd30 comp rx=0 tx=0).stop 2023-11-24T06:08:07.624 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.580+0000 7fe2fd588700 1 -- 172.21.15.91:0/2498450298 reap_dead start 2023-11-24T06:08:07.624 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.580+0000 7fe2fefeb700 1 -- 172.21.15.91:0/2498450298 shutdown_connections 2023-11-24T06:08:07.624 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.580+0000 7fe2fefeb700 1 -- 172.21.15.91:0/2498450298 >> 172.21.15.91:0/2498450298 conn(0x7fe2f80cf980 msgr2=0x7fe2f80ceef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.624 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.580+0000 7fe2fefeb700 1 -- 172.21.15.91:0/2498450298 shutdown_connections 2023-11-24T06:08:07.624 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.580+0000 7fe2fefeb700 1 -- 172.21.15.91:0/2498450298 wait complete. 2023-11-24T06:08:07.624 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.581+0000 7f8d82f04700 1 -- 172.21.15.91:0/4284397577 --> [v2:172.21.15.91:6808/649116497,v1:172.21.15.91:6809/649116497] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8d5c005ca0 con 0x7f8d5c0015d0 2023-11-24T06:08:07.625 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.573+0000 7f7f0e1a7700 1 -- 172.21.15.91:0/2318914280 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f0806cae0 msgr2=0x7f7f080ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.625 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.573+0000 7f7f0e1a7700 1 --2- 172.21.15.91:0/2318914280 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f0806cae0 0x7f7f080ce440 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00aa90 tx=0x7f7efc009360 comp rx=0 tx=0).stop 2023-11-24T06:08:07.625 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.582+0000 7f8d797fa700 1 -- 172.21.15.91:0/4284397577 <== osd.2 v2:172.21.15.91:6808/649116497 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8d5c005ca0 con 0x7f8d5c0015d0 2023-11-24T06:08:07.626 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.582+0000 7f8d82f04700 1 -- 172.21.15.91:0/4284397577 >> [v2:172.21.15.91:6808/649116497,v1:172.21.15.91:6809/649116497] conn(0x7f8d5c0015d0 msgr2=0x7f8d5c003a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.626 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.582+0000 7f8d82f04700 1 --2- 172.21.15.91:0/4284397577 >> [v2:172.21.15.91:6808/649116497,v1:172.21.15.91:6809/649116497] conn(0x7f8d5c0015d0 0x7f8d5c003a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.626 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.582+0000 7f8d82f04700 1 -- 172.21.15.91:0/4284397577 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8d6c06b450 msgr2=0x7f8d6c06d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.626 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.582+0000 7f8d82f04700 1 --2- 172.21.15.91:0/4284397577 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8d6c06b450 0x7f8d6c06d910 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8d70009b90 tx=0x7f8d70008000 comp rx=0 tx=0).stop 2023-11-24T06:08:07.627 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.582+0000 7f8d82f04700 1 -- 172.21.15.91:0/4284397577 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d7c06f0c0 msgr2=0x7f8d7c12f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.627 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.582+0000 7f8d82f04700 1 --2- 172.21.15.91:0/4284397577 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d7c06f0c0 0x7f8d7c12f900 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8d6800c750 tx=0x7f8d6800cb10 comp rx=0 tx=0).stop 2023-11-24T06:08:07.627 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.582+0000 7f8d814a1700 1 -- 172.21.15.91:0/4284397577 reap_dead start 2023-11-24T06:08:07.627 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.582+0000 7f8d82f04700 1 -- 172.21.15.91:0/4284397577 shutdown_connections 2023-11-24T06:08:07.627 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.582+0000 7f8d82f04700 1 -- 172.21.15.91:0/4284397577 >> 172.21.15.91:0/4284397577 conn(0x7f8d7c0cf980 msgr2=0x7f8d7c0c9fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.627 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.582+0000 7f8d82f04700 1 -- 172.21.15.91:0/4284397577 shutdown_connections 2023-11-24T06:08:07.627 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.582+0000 7f8d82f04700 1 -- 172.21.15.91:0/4284397577 wait complete. 2023-11-24T06:08:07.627 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.584+0000 7f7f0e1a7700 1 -- 172.21.15.91:0/2318914280 shutdown_connections 2023-11-24T06:08:07.627 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.584+0000 7f7f0e1a7700 1 --2- 172.21.15.91:0/2318914280 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7f080ce980 0x7f7f080cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.584+0000 7f7f0e1a7700 1 --2- 172.21.15.91:0/2318914280 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f0806cae0 0x7f7f080ce440 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.584+0000 7f7f0e1a7700 1 -- 172.21.15.91:0/2318914280 >> 172.21.15.91:0/2318914280 conn(0x7f7f080cf980 msgr2=0x7f7f080cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.588+0000 7f2d62b27700 1 -- 172.21.15.91:0/2585874796 --> [v2:172.21.15.94:6824/3888015416,v1:172.21.15.94:6825/3888015416] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2d5c0ceda0 con 0x7f2d40073140 2023-11-24T06:08:07.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.589+0000 7f7f0e1a7700 1 -- 172.21.15.91:0/2318914280 shutdown_connections 2023-11-24T06:08:07.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.589+0000 7ff89d945700 1 --2- 172.21.15.91:0/513294259 >> [v2:172.21.15.94:6808/4188708142,v1:172.21.15.94:6809/4188708142] conn(0x7ff8740015d0 0x7ff874003a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.629 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.589+0000 7ff89d945700 1 -- 172.21.15.91:0/513294259 --> [v2:172.21.15.94:6808/4188708142,v1:172.21.15.94:6809/4188708142] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff874006bc0 con 0x7ff8740015d0 2023-11-24T06:08:07.629 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.589+0000 7f7f0e1a7700 1 -- 172.21.15.91:0/2318914280 wait complete. 2023-11-24T06:08:07.629 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.591+0000 7f7f0e1a7700 1 Processor -- start 2023-11-24T06:08:07.629 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.591+0000 7f7f0e1a7700 1 -- start start 2023-11-24T06:08:07.629 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.592+0000 7f7f0e1a7700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7f0806cae0 0x7f7f080b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.629 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.592+0000 7f7f0e1a7700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f08070a80 0x7f7f080c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.592+0000 7f7f0e1a7700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f080ce980 0x7f7f080c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.592+0000 7f7f0e1a7700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f081565d0 con 0x7f7f08070a80 2023-11-24T06:08:07.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.592+0000 7f7f0e1a7700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0816a130 con 0x7f7f080ce980 2023-11-24T06:08:07.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.592+0000 7f7f0e1a7700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f080c9800 con 0x7f7f0806cae0 2023-11-24T06:08:07.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.592+0000 7f7f06ffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f08070a80 0x7f7f080c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.593+0000 7f7f06ffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f08070a80 0x7f7f080c8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:57946/0 (socket says 172.21.15.91:57946) 2023-11-24T06:08:07.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.593+0000 7f7f07fff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f080ce980 0x7f7f080c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.593+0000 7f7f06ffd700 1 -- 172.21.15.91:0/318754355 learned_addr learned my addr 172.21.15.91:0/318754355 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.631 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.593+0000 7f2d597fa700 1 -- 172.21.15.91:0/2585874796 <== osd.4 v2:172.21.15.94:6824/3888015416 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2d4c004030 con 0x7f2d40073140 2023-11-24T06:08:07.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.593+0000 7f7f077fe700 1 --2- 172.21.15.91:0/318754355 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7f0806cae0 0x7f7f080b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.593+0000 7f7f06ffd700 1 -- 172.21.15.91:0/318754355 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7f0806cae0 msgr2=0x7f7f080b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.593+0000 7f7f06ffd700 1 --2- 172.21.15.91:0/318754355 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7f0806cae0 0x7f7f080b05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.593+0000 7f7f06ffd700 1 -- 172.21.15.91:0/318754355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f080ce980 msgr2=0x7f7f080c92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.593+0000 7f7f06ffd700 1 --2- 172.21.15.91:0/318754355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f080ce980 0x7f7f080c92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.593+0000 7f7f06ffd700 1 -- 172.21.15.91:0/318754355 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc00a6f0 con 0x7f7f08070a80 2023-11-24T06:08:07.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.593+0000 7f7f077fe700 1 --2- 172.21.15.91:0/318754355 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7f0806cae0 0x7f7f080b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:08:07.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.593+0000 7f7f06ffd700 1 --2- 172.21.15.91:0/318754355 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f08070a80 0x7f7f080c8d80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7efc01d000 tx=0x7f7efc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.593+0000 7f7eeffff700 1 -- 172.21.15.91:0/318754355 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc00a290 con 0x7f7f08070a80 2023-11-24T06:08:07.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.593+0000 7f7eeffff700 1 -- 172.21.15.91:0/318754355 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7efc019500 con 0x7f7f08070a80 2023-11-24T06:08:07.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.594+0000 7f7f0e1a7700 1 -- 172.21.15.91:0/318754355 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f081593b0 con 0x7f7f08070a80 2023-11-24T06:08:07.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.594+0000 7f7eeffff700 1 -- 172.21.15.91:0/318754355 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc023690 con 0x7f7f08070a80 2023-11-24T06:08:07.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.594+0000 7f7f0e1a7700 1 -- 172.21.15.91:0/318754355 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f08156a90 con 0x7f7f08070a80 2023-11-24T06:08:07.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.595+0000 7f7eeffff700 1 -- 172.21.15.91:0/318754355 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7efc0237f0 con 0x7f7f08070a80 2023-11-24T06:08:07.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.595+0000 7ff8977fe700 1 --2- 172.21.15.91:0/513294259 >> [v2:172.21.15.94:6808/4188708142,v1:172.21.15.94:6809/4188708142] conn(0x7ff8740015d0 0x7ff874003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.596+0000 7f7f07fff700 1 --2- 172.21.15.91:0/318754355 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f080ce980 0x7f7f080c92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:07.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.596+0000 7f27a62e3700 1 -- 172.21.15.91:0/1767100752 --> [v2:172.21.15.91:6800/1071490516,v1:172.21.15.91:6801/1071490516] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2780002cf0 con 0x7f2788073220 2023-11-24T06:08:07.635 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.596+0000 7f7eeffff700 1 --2- 172.21.15.91:0/318754355 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7ef406aed0 0x7f7ef406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.635 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.596+0000 7ff8977fe700 1 --2- 172.21.15.91:0/513294259 >> [v2:172.21.15.94:6808/4188708142,v1:172.21.15.94:6809/4188708142] conn(0x7ff8740015d0 0x7ff874003a90 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.635 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.596+0000 7f7eeffff700 1 -- 172.21.15.91:0/318754355 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7efc08b570 con 0x7f7f08070a80 2023-11-24T06:08:07.635 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.597+0000 7f7f0e1a7700 1 --2- 172.21.15.91:0/318754355 >> [v2:172.21.15.94:6816/3673570051,v1:172.21.15.94:6817/3673570051] conn(0x7f7f08156c80 0x7f7f08158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.597+0000 7f7f0e1a7700 1 -- 172.21.15.91:0/318754355 --> [v2:172.21.15.94:6816/3673570051,v1:172.21.15.94:6817/3673570051] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f08067f60 con 0x7f7f08156c80 2023-11-24T06:08:07.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.597+0000 7ff887fff700 1 -- 172.21.15.91:0/513294259 <== osd.5 v2:172.21.15.94:6808/4188708142 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff874006bc0 con 0x7ff8740015d0 2023-11-24T06:08:07.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.597+0000 7f279cff9700 1 -- 172.21.15.91:0/1767100752 <== osd.3 v2:172.21.15.91:6800/1071490516 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2780002cf0 con 0x7f2788073220 2023-11-24T06:08:07.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.597+0000 7f7f07fff700 1 --2- 172.21.15.91:0/318754355 >> [v2:172.21.15.94:6816/3673570051,v1:172.21.15.94:6817/3673570051] conn(0x7f7f08156c80 0x7f7f08158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.597+0000 7f7f07fff700 1 --2- 172.21.15.91:0/318754355 >> [v2:172.21.15.94:6816/3673570051,v1:172.21.15.94:6817/3673570051] conn(0x7f7f08156c80 0x7f7f08158660 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.598+0000 7f27a62e3700 1 -- 172.21.15.91:0/1767100752 >> [v2:172.21.15.91:6800/1071490516,v1:172.21.15.91:6801/1071490516] conn(0x7f2788073220 msgr2=0x7f2788075640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.598+0000 7f27a62e3700 1 --2- 172.21.15.91:0/1767100752 >> [v2:172.21.15.91:6800/1071490516,v1:172.21.15.91:6801/1071490516] conn(0x7f2788073220 0x7f2788075640 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.598+0000 7f7f077fe700 1 --2- 172.21.15.91:0/318754355 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7ef406aed0 0x7f7ef406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.598+0000 7f27a62e3700 1 -- 172.21.15.91:0/1767100752 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f278806b080 msgr2=0x7f278806d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.598+0000 7f27a62e3700 1 --2- 172.21.15.91:0/1767100752 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f278806b080 0x7f278806d540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2790006fd0 tx=0x7f2790008040 comp rx=0 tx=0).stop 2023-11-24T06:08:07.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.598+0000 7f27a62e3700 1 -- 172.21.15.91:0/1767100752 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27a014b350 msgr2=0x7f27a00ae5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.598+0000 7f27a62e3700 1 --2- 172.21.15.91:0/1767100752 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27a014b350 0x7f27a00ae5a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2794000c00 tx=0x7f2794004750 comp rx=0 tx=0).stop 2023-11-24T06:08:07.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.598+0000 7f7eeffff700 1 -- 172.21.15.91:0/318754355 <== osd.7 v2:172.21.15.94:6816/3673570051 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f7f08067f60 con 0x7f7f08156c80 2023-11-24T06:08:07.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.598+0000 7f7f077fe700 1 --2- 172.21.15.91:0/318754355 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7ef406aed0 0x7f7ef406d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8007fd0 tx=0x7f7ef8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.598+0000 7f2d62b27700 1 -- 172.21.15.91:0/2585874796 >> [v2:172.21.15.94:6824/3888015416,v1:172.21.15.94:6825/3888015416] conn(0x7f2d40073140 msgr2=0x7f2d40075560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.598+0000 7f2d62b27700 1 --2- 172.21.15.91:0/2585874796 >> [v2:172.21.15.94:6824/3888015416,v1:172.21.15.94:6825/3888015416] conn(0x7f2d40073140 0x7f2d40075560 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.598+0000 7f9ef8f26700 1 -- 172.21.15.91:0/1554500218 --> [v2:172.21.15.91:6813/66406286,v1:172.21.15.91:6815/66406286] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9ef4067f60 con 0x7f9ed4073060 2023-11-24T06:08:07.640 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.599+0000 7f2d62b27700 1 -- 172.21.15.91:0/2585874796 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2d4006af20 msgr2=0x7f2d4006d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.640 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.599+0000 7f2d62b27700 1 --2- 172.21.15.91:0/2585874796 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2d4006af20 0x7f2d4006d3e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2d50006fd0 tx=0x7f2d50009190 comp rx=0 tx=0).stop 2023-11-24T06:08:07.640 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.599+0000 7f2d62b27700 1 -- 172.21.15.91:0/2585874796 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d5c15b170 msgr2=0x7f2d5c13c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.640 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.599+0000 7f2d62b27700 1 --2- 172.21.15.91:0/2585874796 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2d5c15b170 0x7f2d5c13c070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2d5400ba50 tx=0x7f2d5400bd60 comp rx=0 tx=0).stop 2023-11-24T06:08:07.640 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.599+0000 7f2d610c4700 1 -- 172.21.15.91:0/2585874796 reap_dead start 2023-11-24T06:08:07.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.599+0000 7f2d62b27700 1 -- 172.21.15.91:0/2585874796 shutdown_connections 2023-11-24T06:08:07.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.599+0000 7f2d62b27700 1 -- 172.21.15.91:0/2585874796 >> 172.21.15.91:0/2585874796 conn(0x7f2d5c0cf980 msgr2=0x7f2d5c0af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.599+0000 7f2d62b27700 1 -- 172.21.15.91:0/2585874796 shutdown_connections 2023-11-24T06:08:07.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.599+0000 7f2d62b27700 1 -- 172.21.15.91:0/2585874796 wait complete. 2023-11-24T06:08:07.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.599+0000 7fde8f2b4700 1 -- 172.21.15.91:0/4277528224 --> [v2:172.21.15.94:6800/4250977597,v1:172.21.15.94:6801/4250977597] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fde68002d30 con 0x7fde6c073070 2023-11-24T06:08:07.642 INFO:teuthology.orchestra.run.smithi091.stdout:55834574854 2023-11-24T06:08:07.642 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-24T06:08:07.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.600+0000 7f9ee2ffd700 1 -- 172.21.15.91:0/1554500218 <== osd.1 v2:172.21.15.91:6813/66406286 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9ef4067f60 con 0x7f9ed4073060 2023-11-24T06:08:07.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.601+0000 7f27a4880700 1 -- 172.21.15.91:0/1767100752 reap_dead start 2023-11-24T06:08:07.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.601+0000 7f27a62e3700 1 -- 172.21.15.91:0/1767100752 shutdown_connections 2023-11-24T06:08:07.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.601+0000 7f27a62e3700 1 -- 172.21.15.91:0/1767100752 >> 172.21.15.91:0/1767100752 conn(0x7f27a00b1cc0 msgr2=0x7f27a00b0220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.644 INFO:teuthology.orchestra.run.smithi091.stdout:55834574856 2023-11-24T06:08:07.644 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-24T06:08:07.645 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.601+0000 7fde7dffb700 1 -- 172.21.15.91:0/4277528224 <== osd.6 v2:172.21.15.94:6800/4250977597 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fde68002d30 con 0x7fde6c073070 2023-11-24T06:08:07.645 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.601+0000 7f27a62e3700 1 -- 172.21.15.91:0/1767100752 shutdown_connections 2023-11-24T06:08:07.645 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.601+0000 7f27a62e3700 1 -- 172.21.15.91:0/1767100752 wait complete. 2023-11-24T06:08:07.646 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7fde8f2b4700 1 -- 172.21.15.91:0/4277528224 >> [v2:172.21.15.94:6800/4250977597,v1:172.21.15.94:6801/4250977597] conn(0x7fde6c073070 msgr2=0x7fde6c075490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.646 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7fde8f2b4700 1 --2- 172.21.15.91:0/4277528224 >> [v2:172.21.15.94:6800/4250977597,v1:172.21.15.94:6801/4250977597] conn(0x7fde6c073070 0x7fde6c075490 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.646 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7f9ef8f26700 1 -- 172.21.15.91:0/1554500218 >> [v2:172.21.15.91:6813/66406286,v1:172.21.15.91:6815/66406286] conn(0x7f9ed4073060 msgr2=0x7f9ed4075480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.646 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7f9ef8f26700 1 --2- 172.21.15.91:0/1554500218 >> [v2:172.21.15.91:6813/66406286,v1:172.21.15.91:6815/66406286] conn(0x7f9ed4073060 0x7f9ed4075480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.647 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7f9ef8f26700 1 -- 172.21.15.91:0/1554500218 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9ed406aed0 msgr2=0x7f9ed406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.647 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7f9ef8f26700 1 --2- 172.21.15.91:0/1554500218 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9ed406aed0 0x7f9ed406d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8007a70 tx=0x7f9ee8009040 comp rx=0 tx=0).stop 2023-11-24T06:08:07.647 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7f9ef8f26700 1 -- 172.21.15.91:0/1554500218 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9ef41ef6d0 msgr2=0x7f9ef41efb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.647 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7f9ef8f26700 1 --2- 172.21.15.91:0/1554500218 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9ef41ef6d0 0x7f9ef41efb50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9ee400f860 tx=0x7f9ee400fb70 comp rx=0 tx=0).stop 2023-11-24T06:08:07.647 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7f9ef2d9d700 1 -- 172.21.15.91:0/1554500218 reap_dead start 2023-11-24T06:08:07.647 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7fde8f2b4700 1 -- 172.21.15.91:0/4277528224 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fde6c06aed0 msgr2=0x7fde6c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7fde8f2b4700 1 --2- 172.21.15.91:0/4277528224 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fde6c06aed0 0x7fde6c06d390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fde84009ad0 tx=0x7fde84009290 comp rx=0 tx=0).stop 2023-11-24T06:08:07.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7fde8f2b4700 1 -- 172.21.15.91:0/4277528224 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fde8815a520 msgr2=0x7fde880b5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7fde8f2b4700 1 --2- 172.21.15.91:0/4277528224 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fde8815a520 0x7fde880b5380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fde78018000 tx=0x7fde78004750 comp rx=0 tx=0).stop 2023-11-24T06:08:07.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7fde8d851700 1 -- 172.21.15.91:0/4277528224 reap_dead start 2023-11-24T06:08:07.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7fde8f2b4700 1 -- 172.21.15.91:0/4277528224 shutdown_connections 2023-11-24T06:08:07.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7fde8f2b4700 1 -- 172.21.15.91:0/4277528224 >> 172.21.15.91:0/4277528224 conn(0x7fde880cf980 msgr2=0x7fde880b0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7fde8f2b4700 1 -- 172.21.15.91:0/4277528224 shutdown_connections 2023-11-24T06:08:07.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.602+0000 7fde8f2b4700 1 -- 172.21.15.91:0/4277528224 wait complete. 2023-11-24T06:08:07.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.603+0000 7f9ef8f26700 1 -- 172.21.15.91:0/1554500218 shutdown_connections 2023-11-24T06:08:07.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.603+0000 7f9ef8f26700 1 -- 172.21.15.91:0/1554500218 >> 172.21.15.91:0/1554500218 conn(0x7f9ef40cf980 msgr2=0x7f9ef406cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.603+0000 7f9ef8f26700 1 -- 172.21.15.91:0/1554500218 shutdown_connections 2023-11-24T06:08:07.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.603+0000 7f9ef8f26700 1 -- 172.21.15.91:0/1554500218 wait complete. 2023-11-24T06:08:07.650 INFO:teuthology.orchestra.run.smithi091.stdout:55834574855 2023-11-24T06:08:07.650 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-24T06:08:07.651 INFO:teuthology.orchestra.run.smithi091.stdout:55834574856 2023-11-24T06:08:07.651 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-24T06:08:07.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.618+0000 7f7f0e1a7700 1 -- 172.21.15.91:0/318754355 --> [v2:172.21.15.94:6816/3673570051,v1:172.21.15.94:6817/3673570051] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7f080cf500 con 0x7f7f08156c80 2023-11-24T06:08:07.652 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.618+0000 7f7eeffff700 1 -- 172.21.15.91:0/318754355 <== osd.7 v2:172.21.15.94:6816/3673570051 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7f080cf500 con 0x7f7f08156c80 2023-11-24T06:08:07.652 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.619+0000 7f7f0e1a7700 1 -- 172.21.15.91:0/318754355 >> [v2:172.21.15.94:6816/3673570051,v1:172.21.15.94:6817/3673570051] conn(0x7f7f08156c80 msgr2=0x7f7f08158660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.619+0000 7f7f0e1a7700 1 --2- 172.21.15.91:0/318754355 >> [v2:172.21.15.94:6816/3673570051,v1:172.21.15.94:6817/3673570051] conn(0x7f7f08156c80 0x7f7f08158660 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.619+0000 7f7f0e1a7700 1 -- 172.21.15.91:0/318754355 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7ef406aed0 msgr2=0x7f7ef406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.619+0000 7f7f0e1a7700 1 --2- 172.21.15.91:0/318754355 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7ef406aed0 0x7f7ef406d390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8007fd0 tx=0x7f7ef8009040 comp rx=0 tx=0).stop 2023-11-24T06:08:07.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.619+0000 7f7f0e1a7700 1 -- 172.21.15.91:0/318754355 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f08070a80 msgr2=0x7f7f080c8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.619+0000 7f7f0e1a7700 1 --2- 172.21.15.91:0/318754355 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f08070a80 0x7f7f080c8d80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7efc01d000 tx=0x7f7efc004750 comp rx=0 tx=0).stop 2023-11-24T06:08:07.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.619+0000 7f7f07fff700 1 -- 172.21.15.91:0/318754355 reap_dead start 2023-11-24T06:08:07.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.619+0000 7f7f0e1a7700 1 -- 172.21.15.91:0/318754355 shutdown_connections 2023-11-24T06:08:07.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.619+0000 7f7f0e1a7700 1 -- 172.21.15.91:0/318754355 >> 172.21.15.91:0/318754355 conn(0x7f7f080cf980 msgr2=0x7f7f080cafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.619+0000 7f7f0e1a7700 1 -- 172.21.15.91:0/318754355 shutdown_connections 2023-11-24T06:08:07.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.619+0000 7f7f0e1a7700 1 -- 172.21.15.91:0/318754355 wait complete. 2023-11-24T06:08:07.654 INFO:teuthology.orchestra.run.smithi091.stdout:55834574855 2023-11-24T06:08:07.654 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-24T06:08:07.655 INFO:teuthology.orchestra.run.smithi091.stdout:55834574855 2023-11-24T06:08:07.655 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-24T06:08:07.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.626+0000 7ff89d945700 1 -- 172.21.15.91:0/513294259 --> [v2:172.21.15.94:6808/4188708142,v1:172.21.15.94:6809/4188708142] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff874005ca0 con 0x7ff8740015d0 2023-11-24T06:08:07.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.627+0000 7ff887fff700 1 -- 172.21.15.91:0/513294259 <== osd.5 v2:172.21.15.94:6808/4188708142 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff874005ca0 con 0x7ff8740015d0 2023-11-24T06:08:07.657 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.627+0000 7ff89d945700 1 -- 172.21.15.91:0/513294259 >> [v2:172.21.15.94:6808/4188708142,v1:172.21.15.94:6809/4188708142] conn(0x7ff8740015d0 msgr2=0x7ff874003a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.657 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.627+0000 7ff89d945700 1 --2- 172.21.15.91:0/513294259 >> [v2:172.21.15.94:6808/4188708142,v1:172.21.15.94:6809/4188708142] conn(0x7ff8740015d0 0x7ff874003a90 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.657 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.627+0000 7ff89d945700 1 -- 172.21.15.91:0/513294259 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff87c06af20 msgr2=0x7ff87c06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.657 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.627+0000 7ff89d945700 1 --2- 172.21.15.91:0/513294259 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff87c06af20 0x7ff87c06d3e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff8980cfe50 tx=0x7ff888009040 comp rx=0 tx=0).stop 2023-11-24T06:08:07.657 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.627+0000 7ff89d945700 1 -- 172.21.15.91:0/513294259 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8981448c0 msgr2=0x7ff8980726c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.657 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.627+0000 7ff89d945700 1 --2- 172.21.15.91:0/513294259 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8981448c0 0x7ff8980726c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff88c009fd0 tx=0x7ff88c004750 comp rx=0 tx=0).stop 2023-11-24T06:08:07.658 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.627+0000 7ff8977fe700 1 -- 172.21.15.91:0/513294259 reap_dead start 2023-11-24T06:08:07.658 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.627+0000 7ff89d945700 1 -- 172.21.15.91:0/513294259 shutdown_connections 2023-11-24T06:08:07.658 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.627+0000 7ff89d945700 1 -- 172.21.15.91:0/513294259 >> 172.21.15.91:0/513294259 conn(0x7ff8980b1cc0 msgr2=0x7ff898152970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.658 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.628+0000 7ff89d945700 1 -- 172.21.15.91:0/513294259 shutdown_connections 2023-11-24T06:08:07.658 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.628+0000 7ff89d945700 1 -- 172.21.15.91:0/513294259 wait complete. 2023-11-24T06:08:07.659 INFO:teuthology.orchestra.run.smithi091.stdout:55834574855 2023-11-24T06:08:07.659 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-24T06:08:07.660 INFO:teuthology.orchestra.run.smithi091.stdout:55834574856 2023-11-24T06:08:07.660 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-24T06:08:07.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.887+0000 7f11a08f2700 1 Processor -- start 2023-11-24T06:08:07.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.887+0000 7f11a08f2700 1 -- start start 2023-11-24T06:08:07.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.888+0000 7f0823088700 1 Processor -- start 2023-11-24T06:08:07.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.888+0000 7f0823088700 1 -- start start 2023-11-24T06:08:07.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.889+0000 7f11a08f2700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f119c15db10 0x7f119c15def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.889+0000 7f11a08f2700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f119c0ca2c0 0x7f119c0ca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.889+0000 7f11a08f2700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f119c0ce000 con 0x7f119c071390 2023-11-24T06:08:07.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.889+0000 7f11a08f2700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f119c0ce2c0 con 0x7f119c0ca2c0 2023-11-24T06:08:07.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.889+0000 7f11a08f2700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f119c0ce5a0 con 0x7f119c15db10 2023-11-24T06:08:07.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.889+0000 7f0823088700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f081c0cbd90 0x7f081c1670b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.889+0000 7f0823088700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081c0cb2d0 con 0x7f081c071390 2023-11-24T06:08:07.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.889+0000 7f0823088700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081c0ce2c0 con 0x7f081c0d04a0 2023-11-24T06:08:07.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.889+0000 7f0823088700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081c0ce5a0 con 0x7f081c0cbd90 2023-11-24T06:08:07.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.889+0000 7f119a59c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f119c15db10 0x7f119c15def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.890+0000 7f119a59c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f119c15db10 0x7f119c15def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:41386/0 (socket says 172.21.15.91:41386) 2023-11-24T06:08:07.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.890+0000 7f119a59c700 1 -- 172.21.15.91:0/2683792442 learned_addr learned my addr 172.21.15.91:0/2683792442 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.890+0000 7f081bfff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f081c0d04a0 0x7f081c15ac30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.91:54686/0 (socket says 172.21.15.91:54686) 2023-11-24T06:08:07.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.890+0000 7f081bfff700 1 -- 172.21.15.91:0/104941672 learned_addr learned my addr 172.21.15.91:0/104941672 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:07.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.890+0000 7f0821625700 1 --2- 172.21.15.91:0/104941672 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f081c0cbd90 0x7f081c1670b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.890+0000 7f119a59c700 1 -- 172.21.15.91:0/2683792442 >> v2:172.21.15.91:3300/0 conn(0x7f119c0ca2c0 msgr2=0x7f119c0ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.890+0000 7f119a59c700 1 --2- 172.21.15.91:0/2683792442 >> v2:172.21.15.91:3300/0 conn(0x7f119c0ca2c0 0x7f119c0ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.890+0000 7f119a59c700 1 -- 172.21.15.91:0/2683792442 >> v1:172.21.15.91:6789/0 conn(0x7f119c071390 legacy=0x7f119c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.890+0000 7f119a59c700 1 -- 172.21.15.91:0/2683792442 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f119c0ce880 con 0x7f119c15db10 2023-11-24T06:08:07.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.890+0000 7f081b7fe700 1 -- 172.21.15.91:0/104941672 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161574738 0 0) 0x7f081c0ce2c0 con 0x7f081c0d04a0 2023-11-24T06:08:07.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.890+0000 7f081b7fe700 1 -- 172.21.15.91:0/104941672 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08040035b0 con 0x7f081c0d04a0 2023-11-24T06:08:07.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.890+0000 7f119a59c700 1 --2- 172.21.15.91:0/2683792442 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f119c15db10 0x7f119c15def0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f118c0092e0 tx=0x7f118c0095f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=537e532247100bb7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.890+0000 7f119959a700 1 -- 172.21.15.91:0/2683792442 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c004230 con 0x7f119c15db10 2023-11-24T06:08:07.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.890+0000 7f119959a700 1 -- 172.21.15.91:0/2683792442 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118c004390 con 0x7f119c15db10 2023-11-24T06:08:07.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.890+0000 7f119959a700 1 -- 172.21.15.91:0/2683792442 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118c00c770 con 0x7f119c15db10 2023-11-24T06:08:07.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7f081b7fe700 1 -- 172.21.15.91:0/104941672 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 790804992 0 0) 0x7f08040035b0 con 0x7f081c0d04a0 2023-11-24T06:08:07.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7f081b7fe700 1 -- 172.21.15.91:0/104941672 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f081c0cbd90 msgr2=0x7f081c1670b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7f081b7fe700 1 --2- 172.21.15.91:0/104941672 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f081c0cbd90 0x7f081c1670b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7f081b7fe700 1 -- 172.21.15.91:0/104941672 >> v1:172.21.15.91:6789/0 conn(0x7f081c071390 legacy=0x7f081c15a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7f081b7fe700 1 -- 172.21.15.91:0/104941672 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081c0ce880 con 0x7f081c0d04a0 2023-11-24T06:08:07.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7f081b7fe700 1 -- 172.21.15.91:0/104941672 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f080c001670 con 0x7f081c0d04a0 2023-11-24T06:08:07.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7f081b7fe700 1 -- 172.21.15.91:0/104941672 >> v1:172.21.15.94:6789/0 conn(0x7f081c0d04a0 legacy=0x7f081c15ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7f081b7fe700 1 --2- 172.21.15.91:0/104941672 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0804003a30 0x7f0804003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7f081b7fe700 1 --2- 172.21.15.91:0/104941672 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0804004850 0x7f0804004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7f081b7fe700 1 --2- 172.21.15.91:0/104941672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0804005210 0x7f080400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7f081b7fe700 1 -- 172.21.15.91:0/104941672 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081c0ce5a0 con 0x7f0804003a30 2023-11-24T06:08:07.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7f081b7fe700 1 -- 172.21.15.91:0/104941672 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080400c0f0 con 0x7f0804005210 2023-11-24T06:08:07.901 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7f081b7fe700 1 -- 172.21.15.91:0/104941672 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080400c2b0 con 0x7f0804004850 2023-11-24T06:08:07.901 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7f0820e24700 1 --2- 172.21.15.91:0/104941672 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0804003a30 0x7f0804003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.901 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7f081bfff700 1 --2- 172.21.15.91:0/104941672 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0804004850 0x7f0804004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.901 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7ffa7a7fb700 1 Processor -- start 2023-11-24T06:08:07.901 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7fe395f4e700 1 Processor -- start 2023-11-24T06:08:07.902 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7f11a08f2700 1 -- 172.21.15.91:0/2683792442 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f119c15db10 msgr2=0x7f119c15def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.902 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7f11a08f2700 1 --2- 172.21.15.91:0/2683792442 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f119c15db10 0x7f119c15def0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f118c0092e0 tx=0x7f118c0095f0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.902 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7f11a08f2700 1 -- 172.21.15.91:0/2683792442 shutdown_connections 2023-11-24T06:08:07.902 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7f11a08f2700 1 --2- 172.21.15.91:0/2683792442 >> v2:172.21.15.91:3300/0 conn(0x7f119c0ca2c0 0x7f119c0ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.902 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7f11a08f2700 1 --2- 172.21.15.91:0/2683792442 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f119c15db10 0x7f119c15def0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.903 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7f11a08f2700 1 -- 172.21.15.91:0/2683792442 >> 172.21.15.91:0/2683792442 conn(0x7f119c0cf980 msgr2=0x7f119c0cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.903 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.893+0000 7f11a08f2700 1 -- 172.21.15.91:0/2683792442 shutdown_connections 2023-11-24T06:08:07.903 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.893+0000 7f11a08f2700 1 -- 172.21.15.91:0/2683792442 wait complete. 2023-11-24T06:08:07.903 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7f081bfff700 1 -- 172.21.15.91:0/104941672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0804005210 msgr2=0x7f080400ba10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-24T06:08:07.903 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7f081bfff700 1 --2- 172.21.15.91:0/104941672 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0804005210 0x7f080400ba10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.904 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7f081bfff700 1 -- 172.21.15.91:0/104941672 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0804003a30 msgr2=0x7f0804003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.904 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7f081bfff700 1 --2- 172.21.15.91:0/104941672 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0804003a30 0x7f0804003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.904 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7f081bfff700 1 -- 172.21.15.91:0/104941672 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08040035b0 con 0x7f0804004850 2023-11-24T06:08:07.905 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7fe395f4e700 1 -- start start 2023-11-24T06:08:07.905 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.893+0000 7fe395f4e700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe39014c040 0x7fe3901629b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.905 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.893+0000 7fe395f4e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe39006e520 0x7fe39006afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.905 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.893+0000 7fe395f4e700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3900b48d0 con 0x7fe39006e9a0 2023-11-24T06:08:07.905 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.893+0000 7fe395f4e700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe390162ef0 con 0x7fe39014c040 2023-11-24T06:08:07.905 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.893+0000 7fe395f4e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39006e250 con 0x7fe39006e520 2023-11-24T06:08:07.906 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.893+0000 7fe38f7fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe39014c040 0x7fe3901629b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.906 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.893+0000 7fe38f7fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe39014c040 0x7fe3901629b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:42368/0 (socket says 172.21.15.91:42368) 2023-11-24T06:08:07.906 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.893+0000 7fe38f7fe700 1 -- 172.21.15.91:0/4221921496 learned_addr learned my addr 172.21.15.91:0/4221921496 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.906 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.893+0000 7fe38f7fe700 1 -- 172.21.15.91:0/4221921496 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe39006e520 msgr2=0x7fe39006afd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T06:08:07.906 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.893+0000 7fe38f7fe700 1 --2- 172.21.15.91:0/4221921496 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe39006e520 0x7fe39006afd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.906 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.893+0000 7fe38f7fe700 1 -- 172.21.15.91:0/4221921496 >> v1:172.21.15.91:6789/0 conn(0x7fe39006e9a0 legacy=0x7fe3900b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.906 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.893+0000 7fe38f7fe700 1 -- 172.21.15.91:0/4221921496 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39006dc90 con 0x7fe39014c040 2023-11-24T06:08:07.906 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.894+0000 7fe38f7fe700 1 --2- 172.21.15.91:0/4221921496 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe39014c040 0x7fe3901629b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe3800098f0 tx=0x7fe380009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f445cdb3d4745ebb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.906 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.894+0000 7fe38e7fc700 1 -- 172.21.15.91:0/4221921496 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380004230 con 0x7fe39014c040 2023-11-24T06:08:07.907 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.894+0000 7fe38e7fc700 1 -- 172.21.15.91:0/4221921496 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe380004390 con 0x7fe39014c040 2023-11-24T06:08:07.907 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.894+0000 7fe38e7fc700 1 -- 172.21.15.91:0/4221921496 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3800036a0 con 0x7fe39014c040 2023-11-24T06:08:07.907 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.895+0000 7f11a08f2700 1 Processor -- start 2023-11-24T06:08:07.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.891+0000 7ffa7a7fb700 1 -- start start 2023-11-24T06:08:07.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7ffa7a7fb700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ffa740cbb80 0x7ffa74070a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7ffa7a7fb700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa7406d1f0 con 0x7ffa7414bf10 2023-11-24T06:08:07.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7ffa7a7fb700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa740cc080 con 0x7ffa74165f70 2023-11-24T06:08:07.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.892+0000 7ffa7a7fb700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa740cdf60 con 0x7ffa740cbb80 2023-11-24T06:08:07.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.895+0000 7f081bfff700 1 --2- 172.21.15.91:0/104941672 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0804004850 0x7f0804004cd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f080c00c190 tx=0x7f0804005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed52e7d3069bc190 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.895+0000 7f081b7fe700 1 -- 172.21.15.91:0/104941672 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080c00cd30 con 0x7f0804004850 2023-11-24T06:08:07.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.895+0000 7fe395f4e700 1 -- 172.21.15.91:0/4221921496 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe39014c040 msgr2=0x7fe3901629b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.895+0000 7fe395f4e700 1 --2- 172.21.15.91:0/4221921496 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe39014c040 0x7fe3901629b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe3800098f0 tx=0x7fe380009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:07.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.895+0000 7fe395f4e700 1 -- 172.21.15.91:0/4221921496 shutdown_connections 2023-11-24T06:08:07.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.895+0000 7fe395f4e700 1 --2- 172.21.15.91:0/4221921496 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe39014c040 0x7fe3901629b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.895+0000 7fe395f4e700 1 --2- 172.21.15.91:0/4221921496 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe39006e520 0x7fe39006afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.895+0000 7fe395f4e700 1 -- 172.21.15.91:0/4221921496 >> 172.21.15.91:0/4221921496 conn(0x7fe3900d0bc0 msgr2=0x7fe3900d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.895+0000 7fe395f4e700 1 -- 172.21.15.91:0/4221921496 shutdown_connections 2023-11-24T06:08:07.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.895+0000 7ffa78d98700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ffa74165f70 0x7ffa740ce3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.91:54698/0 (socket says 172.21.15.91:54698) 2023-11-24T06:08:07.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.895+0000 7ffa78d98700 1 -- 172.21.15.91:0/3105857783 learned_addr learned my addr 172.21.15.91:0/3105857783 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:07.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.895+0000 7fe395f4e700 1 -- 172.21.15.91:0/4221921496 wait complete. 2023-11-24T06:08:07.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.895+0000 7f11a08f2700 1 -- start start 2023-11-24T06:08:07.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa72ffd700 1 -- 172.21.15.91:0/3105857783 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946714309 0 0) 0x7ffa740cc080 con 0x7ffa74165f70 2023-11-24T06:08:07.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa72ffd700 1 -- 172.21.15.91:0/3105857783 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa5c0035b0 con 0x7ffa74165f70 2023-11-24T06:08:07.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7f11a08f2700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f119c071390 0x7f119c1e3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7f11a08f2700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f119c0ca2c0 0x7f119c1e6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7f11a08f2700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f119c15db10 0x7f119c200110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7f11a08f2700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119c1e55f0 con 0x7f119c0ca2c0 2023-11-24T06:08:07.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7f11a08f2700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119c1e4550 con 0x7f119c071390 2023-11-24T06:08:07.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7f11a08f2700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f119c1e4830 con 0x7f119c15db10 2023-11-24T06:08:07.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7f119ad9d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f119c15db10 0x7f119c200110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7f119ad9d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f119c15db10 0x7f119c200110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:41418/0 (socket says 172.21.15.91:41418) 2023-11-24T06:08:07.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7f119ad9d700 1 -- 172.21.15.91:0/3238244695 learned_addr learned my addr 172.21.15.91:0/3238244695 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa72ffd700 1 -- 172.21.15.91:0/3105857783 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2199967972 0 0) 0x7ffa5c0035b0 con 0x7ffa74165f70 2023-11-24T06:08:07.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa72ffd700 1 -- 172.21.15.91:0/3105857783 >> v2:172.21.15.91:3300/0 conn(0x7ffa740cbb80 msgr2=0x7ffa74070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa72ffd700 1 --2- 172.21.15.91:0/3105857783 >> v2:172.21.15.91:3300/0 conn(0x7ffa740cbb80 0x7ffa74070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa72ffd700 1 -- 172.21.15.91:0/3105857783 >> v1:172.21.15.91:6789/0 conn(0x7ffa7414bf10 legacy=0x7ffa7406cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa72ffd700 1 -- 172.21.15.91:0/3105857783 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7406e950 con 0x7ffa74165f70 2023-11-24T06:08:07.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa72ffd700 1 -- 172.21.15.91:0/3105857783 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7ffa68001670 con 0x7ffa74165f70 2023-11-24T06:08:07.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa72ffd700 1 -- 172.21.15.91:0/3105857783 >> v1:172.21.15.94:6789/0 conn(0x7ffa74165f70 legacy=0x7ffa740ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa72ffd700 1 --2- 172.21.15.91:0/3105857783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa5c003a00 0x7ffa5c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa72ffd700 1 --2- 172.21.15.91:0/3105857783 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa5c004850 0x7ffa5c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa72ffd700 1 --2- 172.21.15.91:0/3105857783 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ffa5c005330 0x7ffa5c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa72ffd700 1 -- 172.21.15.91:0/3105857783 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5c00c200 con 0x7ffa5c005330 2023-11-24T06:08:07.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa72ffd700 1 -- 172.21.15.91:0/3105857783 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5c00c370 con 0x7ffa5c003a00 2023-11-24T06:08:07.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa72ffd700 1 -- 172.21.15.91:0/3105857783 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5c00c530 con 0x7ffa5c004850 2023-11-24T06:08:07.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa737fe700 1 --2- 172.21.15.91:0/3105857783 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa5c004850 0x7ffa5c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7f119ad9d700 1 -- 172.21.15.91:0/3238244695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f119c071390 msgr2=0x7f119c1e3c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:07.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7f119ad9d700 1 --2- 172.21.15.91:0/3238244695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f119c071390 0x7f119c1e3c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7f119ad9d700 1 -- 172.21.15.91:0/3238244695 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f119c0ca2c0 msgr2=0x7f119c1e6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7f119ad9d700 1 --2- 172.21.15.91:0/3238244695 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f119c0ca2c0 0x7f119c1e6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7f119ad9d700 1 -- 172.21.15.91:0/3238244695 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c009de0 con 0x7f119c15db10 2023-11-24T06:08:07.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.896+0000 7ffa73fff700 1 --2- 172.21.15.91:0/3105857783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa5c003a00 0x7ffa5c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.897+0000 7f081b7fe700 1 -- 172.21.15.91:0/104941672 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080c00ce90 con 0x7f0804004850 2023-11-24T06:08:07.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.897+0000 7f081b7fe700 1 -- 172.21.15.91:0/104941672 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080c005b60 con 0x7f0804004850 2023-11-24T06:08:07.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.897+0000 7f119ad9d700 1 --2- 172.21.15.91:0/3238244695 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f119c15db10 0x7f119c200110 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f119000b550 tx=0x7f119000b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.897+0000 7fe395f4e700 1 Processor -- start 2023-11-24T06:08:07.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.897+0000 7f118affd700 1 -- 172.21.15.91:0/3238244695 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1190011830 con 0x7f119c15db10 2023-11-24T06:08:07.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.897+0000 7f118affd700 1 -- 172.21.15.91:0/3238244695 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1190011990 con 0x7f119c15db10 2023-11-24T06:08:07.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.897+0000 7f118affd700 1 -- 172.21.15.91:0/3238244695 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1190011d40 con 0x7f119c15db10 2023-11-24T06:08:07.918 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.897+0000 7f0823088700 1 -- 172.21.15.91:0/104941672 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0804004850 msgr2=0x7f0804004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.918 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.897+0000 7f0823088700 1 --2- 172.21.15.91:0/104941672 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0804004850 0x7f0804004cd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f080c00c190 tx=0x7f0804005880 comp rx=0 tx=0).stop 2023-11-24T06:08:07.918 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.897+0000 7f0821625700 1 -- 172.21.15.91:0/104941672 reap_dead start 2023-11-24T06:08:07.918 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7f0823088700 1 -- 172.21.15.91:0/104941672 shutdown_connections 2023-11-24T06:08:07.918 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7f0823088700 1 -- 172.21.15.91:0/104941672 >> 172.21.15.91:0/104941672 conn(0x7f081c0cf980 msgr2=0x7f081c0cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.919 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7ffa737fe700 1 -- 172.21.15.91:0/3105857783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa5c003a00 msgr2=0x7ffa5c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.919 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7ffa737fe700 1 --2- 172.21.15.91:0/3105857783 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa5c003a00 0x7ffa5c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.919 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7ffa737fe700 1 -- 172.21.15.91:0/3105857783 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ffa5c005330 msgr2=0x7ffa5c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.919 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7ffa737fe700 1 --2- 172.21.15.91:0/3105857783 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ffa5c005330 0x7ffa5c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.919 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7ffa737fe700 1 -- 172.21.15.91:0/3105857783 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7406e950 con 0x7ffa5c004850 2023-11-24T06:08:07.919 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7ffa78d98700 1 -- 172.21.15.91:0/3105857783 reap_dead start 2023-11-24T06:08:07.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7f11a08f2700 1 -- 172.21.15.91:0/3238244695 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f119c1e4b10 con 0x7f119c15db10 2023-11-24T06:08:07.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7f11a08f2700 1 -- 172.21.15.91:0/3238244695 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f119c1e50d0 con 0x7f119c15db10 2023-11-24T06:08:07.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7ffa737fe700 1 --2- 172.21.15.91:0/3105857783 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa5c004850 0x7ffa5c004cd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffa740cbfb0 tx=0x7ffa5c005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21860683c613d232 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.921 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.897+0000 7fe395f4e700 1 -- start start 2023-11-24T06:08:07.921 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7fe395f4e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe39006e520 0x7fe390138b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.921 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7fe395f4e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe39006e9a0 0x7fe390135290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.921 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7fe395f4e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe39014c040 0x7fe39013c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.921 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7fe395f4e700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe390139080 con 0x7fe39006e9a0 2023-11-24T06:08:07.922 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7fe395f4e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe390136030 con 0x7fe39006e520 2023-11-24T06:08:07.922 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.898+0000 7fe395f4e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe390135d50 con 0x7fe39014c040 2023-11-24T06:08:07.922 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.899+0000 7fe38f7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe39006e520 0x7fe390138b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.922 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.899+0000 7fe38f7fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe39006e520 0x7fe390138b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:42388/0 (socket says 172.21.15.91:42388) 2023-11-24T06:08:07.922 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.899+0000 7fe38f7fe700 1 -- 172.21.15.91:0/2673177771 learned_addr learned my addr 172.21.15.91:0/2673177771 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.922 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.899+0000 7fe38ffff700 1 --2- 172.21.15.91:0/2673177771 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe39014c040 0x7fe39013c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.922 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.899+0000 7ffa72ffd700 1 -- 172.21.15.91:0/3105857783 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa64004230 con 0x7ffa5c004850 2023-11-24T06:08:07.923 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.899+0000 7fe38f7fe700 1 -- 172.21.15.91:0/2673177771 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe39014c040 msgr2=0x7fe39013c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.923 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.899+0000 7fe38f7fe700 1 --2- 172.21.15.91:0/2673177771 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe39014c040 0x7fe39013c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.923 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.899+0000 7fe38f7fe700 1 -- 172.21.15.91:0/2673177771 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe39006e9a0 msgr2=0x7fe390135290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.923 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.899+0000 7fe38f7fe700 1 --2- 172.21.15.91:0/2673177771 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe39006e9a0 0x7fe390135290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.923 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.899+0000 7fe38f7fe700 1 -- 172.21.15.91:0/2673177771 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe380009550 con 0x7fe39006e520 2023-11-24T06:08:07.923 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.899+0000 7fe38f7fe700 1 --2- 172.21.15.91:0/2673177771 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe39006e520 0x7fe390138b40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe380000c00 tx=0x7fe380004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.923 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.900+0000 7f11a08f2700 1 -- 172.21.15.91:0/3238244695 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f119c067f60 con 0x7f119c15db10 2023-11-24T06:08:07.924 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.899+0000 7fe377fff700 1 -- 172.21.15.91:0/2673177771 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38000b660 con 0x7fe39006e520 2023-11-24T06:08:07.924 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.900+0000 7fe377fff700 1 -- 172.21.15.91:0/2673177771 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38000b7e0 con 0x7fe39006e520 2023-11-24T06:08:07.924 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.900+0000 7fe377fff700 1 -- 172.21.15.91:0/2673177771 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38000bb90 con 0x7fe39006e520 2023-11-24T06:08:07.924 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.899+0000 7ffa72ffd700 1 -- 172.21.15.91:0/3105857783 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa640043b0 con 0x7ffa5c004850 2023-11-24T06:08:07.924 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.899+0000 7ffa72ffd700 1 -- 172.21.15.91:0/3105857783 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa640038c0 con 0x7ffa5c004850 2023-11-24T06:08:07.925 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.900+0000 7ffa7a7fb700 1 -- 172.21.15.91:0/3105857783 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa5c004850 msgr2=0x7ffa5c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.925 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.900+0000 7ffa7a7fb700 1 --2- 172.21.15.91:0/3105857783 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa5c004850 0x7ffa5c004cd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffa740cbfb0 tx=0x7ffa5c005880 comp rx=0 tx=0).stop 2023-11-24T06:08:07.925 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.900+0000 7fe395f4e700 1 -- 172.21.15.91:0/2673177771 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe390138060 con 0x7fe39006e520 2023-11-24T06:08:07.925 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.900+0000 7fe395f4e700 1 -- 172.21.15.91:0/2673177771 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe390138620 con 0x7fe39006e520 2023-11-24T06:08:07.925 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.901+0000 7f4370788700 1 Processor -- start 2023-11-24T06:08:07.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.901+0000 7f118affd700 1 -- 172.21.15.91:0/3238244695 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f119001a400 con 0x7f119c15db10 2023-11-24T06:08:07.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.902+0000 7f118affd700 1 --2- 172.21.15.91:0/3238244695 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f117c06af20 0x7f117c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.902+0000 7f118affd700 1 -- 172.21.15.91:0/3238244695 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1190089550 con 0x7f119c15db10 2023-11-24T06:08:07.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.903+0000 7f119a59c700 1 --2- 172.21.15.91:0/3238244695 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f117c06af20 0x7f117c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.902+0000 7f0823088700 1 -- 172.21.15.91:0/104941672 shutdown_connections 2023-11-24T06:08:07.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.902+0000 7f0823088700 1 -- 172.21.15.91:0/104941672 wait complete. 2023-11-24T06:08:07.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.901+0000 7f4370788700 1 -- start start 2023-11-24T06:08:07.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.902+0000 7f4370788700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f43680cbb80 0x7f4368070a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.902+0000 7f4370788700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436806d1f0 con 0x7f4368165f70 2023-11-24T06:08:07.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.903+0000 7f4370788700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43680cc080 con 0x7f436814bf10 2023-11-24T06:08:07.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.903+0000 7f4370788700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43680cdf60 con 0x7f43680cbb80 2023-11-24T06:08:07.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.902+0000 7fe395f4e700 1 -- 172.21.15.91:0/2673177771 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe390138810 con 0x7fe39006e520 2023-11-24T06:08:07.928 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.903+0000 7f436e524700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f436814bf10 0x7f436806cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.91:54728/0 (socket says 172.21.15.91:54728) 2023-11-24T06:08:07.928 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.903+0000 7f436e524700 1 -- 172.21.15.91:0/3932545344 learned_addr learned my addr 172.21.15.91:0/3932545344 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:07.928 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.903+0000 7f436dd23700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f43680cbb80 0x7f4368070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.928 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.903+0000 7fe377fff700 1 -- 172.21.15.91:0/2673177771 <== mon.1 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe38000bcf0 con 0x7fe39006e520 2023-11-24T06:08:07.928 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.903+0000 7f436dd23700 1 -- 172.21.15.91:0/3932545344 >> v1:172.21.15.94:6789/0 conn(0x7f436814bf10 legacy=0x7f436806cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.929 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.903+0000 7f436dd23700 1 -- 172.21.15.91:0/3932545344 >> v1:172.21.15.91:6789/0 conn(0x7f4368165f70 legacy=0x7f43680ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.929 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.903+0000 7f436dd23700 1 -- 172.21.15.91:0/3932545344 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436806e950 con 0x7f43680cbb80 2023-11-24T06:08:07.929 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.903+0000 7f119a59c700 1 --2- 172.21.15.91:0/3238244695 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f117c06af20 0x7f117c06d3e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f119c141240 tx=0x7f118c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.929 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.903+0000 7f436dd23700 1 --2- 172.21.15.91:0/3932545344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43680cbb80 0x7f4368070a80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4358009840 tx=0x7f4358009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55d0f76d3d05ecc7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.930 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.903+0000 7f0823088700 1 Processor -- start 2023-11-24T06:08:07.930 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.904+0000 7f0823088700 1 -- start start 2023-11-24T06:08:07.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.903+0000 7fe377fff700 1 --2- 172.21.15.91:0/2673177771 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe37c06afa0 0x7fe37c06d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.904+0000 7fe377fff700 1 -- 172.21.15.91:0/2673177771 <== mon.1 v2:172.21.15.94:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe38008ad00 con 0x7fe39006e520 2023-11-24T06:08:07.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.904+0000 7fe38effd700 1 --2- 172.21.15.91:0/2673177771 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe37c06afa0 0x7fe37c06d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.904+0000 7fe38effd700 1 --2- 172.21.15.91:0/2673177771 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe37c06afa0 0x7fe37c06d460 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe384009770 tx=0x7fe384006cd0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7ffa7a7fb700 1 -- 172.21.15.91:0/3105857783 shutdown_connections 2023-11-24T06:08:07.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7ffa7a7fb700 1 --2- 172.21.15.91:0/3105857783 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa5c004850 0x7ffa5c004cd0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7ffa7a7fb700 1 -- 172.21.15.91:0/3105857783 >> 172.21.15.91:0/3105857783 conn(0x7ffa740cf980 msgr2=0x7ffa740cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7ffa7a7fb700 1 -- 172.21.15.91:0/3105857783 shutdown_connections 2023-11-24T06:08:07.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7ffa7a7fb700 1 -- 172.21.15.91:0/3105857783 wait complete. 2023-11-24T06:08:07.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7f0823088700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0804004850 0x7f081c156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7f0823088700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f081c159800 0x7f081c156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7f0823088700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f081c159290 0x7f081c13c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7f0823088700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081c13c4f0 con 0x7f0804004850 2023-11-24T06:08:07.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7f0823088700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081c13dd40 con 0x7f081c159290 2023-11-24T06:08:07.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7f0823088700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f081c13e020 con 0x7f081c159800 2023-11-24T06:08:07.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7f0820e24700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0804004850 0x7f081c156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7f0820e24700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0804004850 0x7f081c156d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58010/0 (socket says 172.21.15.91:58010) 2023-11-24T06:08:07.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7f0820e24700 1 -- 172.21.15.91:0/3299192464 learned_addr learned my addr 172.21.15.91:0/3299192464 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7f081bfff700 1 --2- 172.21.15.91:0/3299192464 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f081c159800 0x7f081c156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7f0820e24700 1 -- 172.21.15.91:0/3299192464 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f081c159800 msgr2=0x7f081c156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7f0820e24700 1 --2- 172.21.15.91:0/3299192464 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f081c159800 0x7f081c156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7f0820e24700 1 -- 172.21.15.91:0/3299192464 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f081c159290 msgr2=0x7f081c13c980 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:07.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7f0820e24700 1 --2- 172.21.15.91:0/3299192464 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f081c159290 0x7f081c13c980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.905+0000 7f0820e24700 1 -- 172.21.15.91:0/3299192464 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081c13e300 con 0x7f0804004850 2023-11-24T06:08:07.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.906+0000 7f0820e24700 1 --2- 172.21.15.91:0/3299192464 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0804004850 0x7f081c156d40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f081000e5d0 tx=0x7f081000e8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.906+0000 7f08197fa700 1 -- 172.21.15.91:0/3299192464 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08100052e0 con 0x7f0804004850 2023-11-24T06:08:07.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.906+0000 7f0823088700 1 -- 172.21.15.91:0/3299192464 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f081c13bea0 con 0x7f0804004850 2023-11-24T06:08:07.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.906+0000 7f0823088700 1 -- 172.21.15.91:0/3299192464 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f081c0b4440 con 0x7f0804004850 2023-11-24T06:08:07.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.906+0000 7f08197fa700 1 -- 172.21.15.91:0/3299192464 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0810005440 con 0x7f0804004850 2023-11-24T06:08:07.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.906+0000 7f08197fa700 1 -- 172.21.15.91:0/3299192464 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0810003880 con 0x7f0804004850 2023-11-24T06:08:07.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.907+0000 7ffa7a7fb700 1 Processor -- start 2023-11-24T06:08:07.935 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.907+0000 7ffa7a7fb700 1 -- start start 2023-11-24T06:08:07.935 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.907+0000 7f118affd700 1 -- 172.21.15.91:0/3238244695 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f11900546e0 con 0x7f119c15db10 2023-11-24T06:08:07.935 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.907+0000 7f0823088700 1 -- 172.21.15.91:0/3299192464 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08080052e0 con 0x7f0804004850 2023-11-24T06:08:07.936 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.908+0000 7ffa7a7fb700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ffa5c004850 0x7ffa741457b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.936 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.908+0000 7ffa7a7fb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa74160b00 0x7ffa74162890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.936 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.908+0000 7ffa7a7fb700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa7414a880 0x7ffa7414ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.936 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.908+0000 7ffa7a7fb700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa74163c40 con 0x7ffa5c004850 2023-11-24T06:08:07.936 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.908+0000 7ffa7a7fb700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa74145cf0 con 0x7ffa7414a880 2023-11-24T06:08:07.936 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.908+0000 7ffa7a7fb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa74162dd0 con 0x7ffa74160b00 2023-11-24T06:08:07.936 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.908+0000 7ffa78d98700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa7414a880 0x7ffa7414ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.936 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.908+0000 7ffa737fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa74160b00 0x7ffa74162890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.937 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.908+0000 7ffa78d98700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa7414a880 0x7ffa7414ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:42412/0 (socket says 172.21.15.91:42412) 2023-11-24T06:08:07.937 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.908+0000 7ffa78d98700 1 -- 172.21.15.91:0/1082349052 learned_addr learned my addr 172.21.15.91:0/1082349052 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.937 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.909+0000 7f436d522700 1 -- 172.21.15.91:0/3932545344 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4358011830 con 0x7f43680cbb80 2023-11-24T06:08:07.937 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.909+0000 7f08197fa700 1 -- 172.21.15.91:0/3299192464 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f08100039e0 con 0x7f0804004850 2023-11-24T06:08:07.937 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.909+0000 7f436d522700 1 -- 172.21.15.91:0/3932545344 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4358011990 con 0x7f43680cbb80 2023-11-24T06:08:07.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.909+0000 7f436d522700 1 -- 172.21.15.91:0/3932545344 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4358011d00 con 0x7f43680cbb80 2023-11-24T06:08:07.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.909+0000 7fe377fff700 1 -- 172.21.15.91:0/2673177771 <== mon.1 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe380055e20 con 0x7fe39006e520 2023-11-24T06:08:07.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.909+0000 7f08197fa700 1 --2- 172.21.15.91:0/3299192464 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f080406cbc0 0x7f080406f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.909+0000 7f08197fa700 1 -- 172.21.15.91:0/3299192464 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0810089380 con 0x7f0804004850 2023-11-24T06:08:07.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.910+0000 7f4370788700 1 -- 172.21.15.91:0/3932545344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43680cbb80 msgr2=0x7f4368070a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.910+0000 7f4370788700 1 --2- 172.21.15.91:0/3932545344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43680cbb80 0x7f4368070a80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4358009840 tx=0x7f4358009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:07.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.910+0000 7ffa737fe700 1 -- 172.21.15.91:0/1082349052 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa7414a880 msgr2=0x7ffa7414ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.910+0000 7ffa737fe700 1 --2- 172.21.15.91:0/1082349052 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa7414a880 0x7ffa7414ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.910+0000 7ffa737fe700 1 -- 172.21.15.91:0/1082349052 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ffa5c004850 msgr2=0x7ffa741457b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.910+0000 7ffa737fe700 1 --2- 172.21.15.91:0/1082349052 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ffa5c004850 0x7ffa741457b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.910+0000 7ffa737fe700 1 -- 172.21.15.91:0/1082349052 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa74162fc0 con 0x7ffa74160b00 2023-11-24T06:08:07.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.910+0000 7ffa737fe700 1 --2- 172.21.15.91:0/1082349052 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa74160b00 0x7ffa74162890 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c005830 tx=0x7ffa64004860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.909+0000 7f681472e700 1 Processor -- start 2023-11-24T06:08:07.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.910+0000 7f681472e700 1 -- start start 2023-11-24T06:08:07.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.911+0000 7ffa70ff9700 1 -- 172.21.15.91:0/1082349052 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa64011a70 con 0x7ffa74160b00 2023-11-24T06:08:07.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.911+0000 7ffa70ff9700 1 -- 172.21.15.91:0/1082349052 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa64011bd0 con 0x7ffa74160b00 2023-11-24T06:08:07.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.911+0000 7ffa70ff9700 1 -- 172.21.15.91:0/1082349052 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa64023410 con 0x7ffa74160b00 2023-11-24T06:08:07.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.911+0000 7ffa7a7fb700 1 -- 172.21.15.91:0/1082349052 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa741632a0 con 0x7ffa74160b00 2023-11-24T06:08:07.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.911+0000 7ffa7a7fb700 1 -- 172.21.15.91:0/1082349052 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa74163860 con 0x7ffa74160b00 2023-11-24T06:08:07.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.911+0000 7f681472e700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f680c06cae0 0x7f680c16dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.911+0000 7f681472e700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680c0cab10 con 0x7f680c071190 2023-11-24T06:08:07.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.911+0000 7f681472e700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680c0ca1e0 con 0x7f680c0ce440 2023-11-24T06:08:07.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.911+0000 7f681472e700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f680c0c9fd0 con 0x7f680c06cae0 2023-11-24T06:08:07.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.911+0000 7f68124ca700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f680c06cae0 0x7f680c16dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.911+0000 7f68124ca700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f680c06cae0 0x7f680c16dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:42416/0 (socket says 172.21.15.91:42416) 2023-11-24T06:08:07.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.911+0000 7f68124ca700 1 -- 172.21.15.91:0/3131691669 learned_addr learned my addr 172.21.15.91:0/3131691669 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.943 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.912+0000 7f68114c8700 1 -- 172.21.15.91:0/3131691669 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399078301 0 0) 0x7f680c0ca1e0 con 0x7f680c0ce440 2023-11-24T06:08:07.943 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.912+0000 7f68114c8700 1 -- 172.21.15.91:0/3131691669 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f80035b0 con 0x7f680c0ce440 2023-11-24T06:08:07.943 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.912+0000 7f081bfff700 1 --2- 172.21.15.91:0/3299192464 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f080406cbc0 0x7f080406f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.912+0000 7f68114c8700 1 -- 172.21.15.91:0/3131691669 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3737284669 0 0) 0x7f67f80035b0 con 0x7f680c0ce440 2023-11-24T06:08:07.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.912+0000 7f68114c8700 1 -- 172.21.15.91:0/3131691669 >> v2:172.21.15.94:3300/0 conn(0x7f680c06cae0 msgr2=0x7f680c16dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.912+0000 7f68114c8700 1 --2- 172.21.15.91:0/3131691669 >> v2:172.21.15.94:3300/0 conn(0x7f680c06cae0 0x7f680c16dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.912+0000 7f68114c8700 1 -- 172.21.15.91:0/3131691669 >> v1:172.21.15.91:6789/0 conn(0x7f680c071190 legacy=0x7f680c0cafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.945 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.912+0000 7f68114c8700 1 -- 172.21.15.91:0/3131691669 --> v1:172.21.15.94:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c0ca4c0 con 0x7f680c0ce440 2023-11-24T06:08:07.945 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7f081bfff700 1 --2- 172.21.15.91:0/3299192464 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f080406cbc0 0x7f080406f080 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f080c015040 tx=0x7f080c007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.946 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.912+0000 7f68114c8700 1 -- 172.21.15.91:0/3131691669 <== mon.1 v1:172.21.15.94:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f67fc001670 con 0x7f680c0ce440 2023-11-24T06:08:07.946 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7f68114c8700 1 -- 172.21.15.91:0/3131691669 >> v1:172.21.15.94:6789/0 conn(0x7f680c0ce440 legacy=0x7f680c070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.946 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7f68114c8700 1 --2- 172.21.15.91:0/3131691669 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f67f8003a00 0x7f67f8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.946 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7f68114c8700 1 --2- 172.21.15.91:0/3131691669 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f67f8004850 0x7f67f8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.946 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7f68114c8700 1 --2- 172.21.15.91:0/3131691669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67f8005210 0x7f67f800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.946 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7f68114c8700 1 -- 172.21.15.91:0/3131691669 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c0c9fd0 con 0x7f67f8003a00 2023-11-24T06:08:07.946 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7f68114c8700 1 -- 172.21.15.91:0/3131691669 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f80148a0 con 0x7f67f8005210 2023-11-24T06:08:07.947 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7f68114c8700 1 -- 172.21.15.91:0/3131691669 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f8014a60 con 0x7f67f8004850 2023-11-24T06:08:07.947 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7f6811cc9700 1 --2- 172.21.15.91:0/3131691669 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f67f8004850 0x7f67f8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.947 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7f6812ccb700 1 --2- 172.21.15.91:0/3131691669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67f8005210 0x7f67f800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.947 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7f6811cc9700 1 -- 172.21.15.91:0/3131691669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67f8005210 msgr2=0x7f67f800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.947 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7f6811cc9700 1 --2- 172.21.15.91:0/3131691669 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f67f8005210 0x7f67f800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.947 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7f6811cc9700 1 -- 172.21.15.91:0/3131691669 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f67f8003a00 msgr2=0x7f67f8003e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-24T06:08:07.947 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7f6811cc9700 1 --2- 172.21.15.91:0/3131691669 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f67f8003a00 0x7f67f8003e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.947 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7f6811cc9700 1 -- 172.21.15.91:0/3131691669 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f80035b0 con 0x7f67f8004850 2023-11-24T06:08:07.947 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7f6812ccb700 1 -- 172.21.15.91:0/3131691669 reap_dead start 2023-11-24T06:08:07.947 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.914+0000 7f6811cc9700 1 --2- 172.21.15.91:0/3131691669 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f67f8004850 0x7f67f8004cd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00c190 tx=0x7f67f8005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c78feed3d6d4a0d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.947 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.914+0000 7f68114c8700 1 -- 172.21.15.91:0/3131691669 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc007b60 con 0x7f67f8004850 2023-11-24T06:08:07.947 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.914+0000 7f68114c8700 1 -- 172.21.15.91:0/3131691669 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67fc00cc30 con 0x7f67f8004850 2023-11-24T06:08:07.947 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.914+0000 7f68114c8700 1 -- 172.21.15.91:0/3131691669 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc006860 con 0x7f67f8004850 2023-11-24T06:08:07.948 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.913+0000 7ffa70ff9700 1 -- 172.21.15.91:0/1082349052 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ffa6401a3f0 con 0x7ffa74160b00 2023-11-24T06:08:07.948 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.914+0000 7ffa70ff9700 1 --2- 172.21.15.91:0/1082349052 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ffa6006afa0 0x7ffa6006d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.948 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.914+0000 7ffa70ff9700 1 -- 172.21.15.91:0/1082349052 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa64089e40 con 0x7ffa74160b00 2023-11-24T06:08:07.948 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.914+0000 7f681472e700 1 -- 172.21.15.91:0/3131691669 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f67f8004850 msgr2=0x7f67f8004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.948 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.914+0000 7f681472e700 1 --2- 172.21.15.91:0/3131691669 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f67f8004850 0x7f67f8004cd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00c190 tx=0x7f67f8005880 comp rx=0 tx=0).stop 2023-11-24T06:08:07.949 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.915+0000 7f681472e700 1 -- 172.21.15.91:0/3131691669 shutdown_connections 2023-11-24T06:08:07.949 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.915+0000 7f681472e700 1 --2- 172.21.15.91:0/3131691669 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f67f8004850 0x7f67f8004cd0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.949 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.915+0000 7f681472e700 1 -- 172.21.15.91:0/3131691669 >> 172.21.15.91:0/3131691669 conn(0x7f680c0cf980 msgr2=0x7f680c0cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.949 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.915+0000 7ffa73fff700 1 --2- 172.21.15.91:0/1082349052 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ffa6006afa0 0x7ffa6006d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.949 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.915+0000 7ffa73fff700 1 --2- 172.21.15.91:0/1082349052 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ffa6006afa0 0x7ffa6006d460 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c001d10 tx=0x7ffa6c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.915+0000 7f681472e700 1 -- 172.21.15.91:0/3131691669 shutdown_connections 2023-11-24T06:08:07.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.916+0000 7f681472e700 1 -- 172.21.15.91:0/3131691669 wait complete. 2023-11-24T06:08:07.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.916+0000 7ffa7a7fb700 1 -- 172.21.15.91:0/1082349052 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa540052e0 con 0x7ffa74160b00 2023-11-24T06:08:07.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.916+0000 7f08197fa700 1 -- 172.21.15.91:0/3299192464 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0810054650 con 0x7f0804004850 2023-11-24T06:08:07.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.917+0000 7f8416277700 1 Processor -- start 2023-11-24T06:08:07.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.917+0000 7f8416277700 1 -- start start 2023-11-24T06:08:07.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.917+0000 7f681472e700 1 Processor -- start 2023-11-24T06:08:07.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.918+0000 7f681472e700 1 -- start start 2023-11-24T06:08:07.952 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.918+0000 7f8416277700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84100c93e0 0x7f84100cd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.952 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.918+0000 7f8416277700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84100718e0 con 0x7f84100ceff0 2023-11-24T06:08:07.952 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.918+0000 7f8416277700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f841015e4a0 con 0x7f84100cda90 2023-11-24T06:08:07.952 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.918+0000 7f8416277700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84100cf3d0 con 0x7f84100c93e0 2023-11-24T06:08:07.952 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f840ffff700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f84100ceff0 0x7f84100c8cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:52970/0 (socket says 172.21.15.91:52970) 2023-11-24T06:08:07.952 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f840ffff700 1 -- 172.21.15.91:0/218484919 learned_addr learned my addr 172.21.15.91:0/218484919 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:07.952 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f840f7fe700 1 --2- 172.21.15.91:0/218484919 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84100c93e0 0x7f84100cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.953 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f681472e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f67f8004850 0x7f680c16cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.953 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f681472e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f680c15c4f0 0x7f680c15c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.953 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f681472e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f680c15a460 0x7f680c15a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.953 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f681472e700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c15ceb0 con 0x7f67f8004850 2023-11-24T06:08:07.953 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f681472e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c15ae20 con 0x7f680c15a460 2023-11-24T06:08:07.953 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f681472e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680c15b0b0 con 0x7f680c15c4f0 2023-11-24T06:08:07.953 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f68124ca700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f67f8004850 0x7f680c16cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.953 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f68124ca700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f67f8004850 0x7f680c16cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58042/0 (socket says 172.21.15.91:58042) 2023-11-24T06:08:07.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f68124ca700 1 -- 172.21.15.91:0/555313099 learned_addr learned my addr 172.21.15.91:0/555313099 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f68124ca700 1 -- 172.21.15.91:0/555313099 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f680c15c4f0 msgr2=0x7f680c15c970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:07.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f68124ca700 1 --2- 172.21.15.91:0/555313099 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f680c15c4f0 0x7f680c15c970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f68124ca700 1 -- 172.21.15.91:0/555313099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f680c15a460 msgr2=0x7f680c15a8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:07.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f68124ca700 1 --2- 172.21.15.91:0/555313099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f680c15a460 0x7f680c15a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f68124ca700 1 -- 172.21.15.91:0/555313099 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c15b390 con 0x7f67f8004850 2023-11-24T06:08:07.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f840f7fe700 1 -- 172.21.15.91:0/218484919 >> v1:172.21.15.94:6789/0 conn(0x7f84100cda90 legacy=0x7f8410161a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f840f7fe700 1 -- 172.21.15.91:0/218484919 >> v1:172.21.15.91:6789/0 conn(0x7f84100ceff0 legacy=0x7f84100c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f840f7fe700 1 -- 172.21.15.91:0/218484919 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84100ce280 con 0x7f84100c93e0 2023-11-24T06:08:07.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.919+0000 7f840f7fe700 1 --2- 172.21.15.91:0/218484919 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84100c93e0 0x7f84100cd550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8400009870 tx=0x7f8400009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba18eef60d7fc130 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.920+0000 7f68124ca700 1 --2- 172.21.15.91:0/555313099 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f67f8004850 0x7f680c16cf00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f680800e7c0 tx=0x7f680800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.916+0000 7f4370788700 1 -- 172.21.15.91:0/3932545344 shutdown_connections 2023-11-24T06:08:07.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.916+0000 7f4370788700 1 --2- 172.21.15.91:0/3932545344 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43680cbb80 0x7f4368070a80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.916+0000 7f4370788700 1 -- 172.21.15.91:0/3932545344 >> 172.21.15.91:0/3932545344 conn(0x7f43680cf980 msgr2=0x7f43680cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.920+0000 7f6802ffd700 1 -- 172.21.15.91:0/555313099 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808013680 con 0x7f67f8004850 2023-11-24T06:08:07.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.920+0000 7f681472e700 1 -- 172.21.15.91:0/555313099 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680c15b670 con 0x7f67f8004850 2023-11-24T06:08:07.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.920+0000 7f681472e700 1 -- 172.21.15.91:0/555313099 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680c15bc30 con 0x7f67f8004850 2023-11-24T06:08:07.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.920+0000 7f6802ffd700 1 -- 172.21.15.91:0/555313099 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68080137e0 con 0x7f67f8004850 2023-11-24T06:08:07.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.920+0000 7f6802ffd700 1 -- 172.21.15.91:0/555313099 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808013b50 con 0x7f67f8004850 2023-11-24T06:08:07.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.920+0000 7f4370788700 1 -- 172.21.15.91:0/3932545344 shutdown_connections 2023-11-24T06:08:07.957 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.920+0000 7f4370788700 1 -- 172.21.15.91:0/3932545344 wait complete. 2023-11-24T06:08:07.957 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.922+0000 7f6802ffd700 1 -- 172.21.15.91:0/555313099 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6808013cb0 con 0x7f67f8004850 2023-11-24T06:08:07.957 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.922+0000 7f6802ffd700 1 --2- 172.21.15.91:0/555313099 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f67f006afa0 0x7f67f006d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.957 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.923+0000 7f6802ffd700 1 -- 172.21.15.91:0/555313099 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6808089510 con 0x7f67f8004850 2023-11-24T06:08:07.957 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.923+0000 7f6811cc9700 1 --2- 172.21.15.91:0/555313099 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f67f006afa0 0x7f67f006d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.958 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.923+0000 7f6811cc9700 1 --2- 172.21.15.91:0/555313099 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f67f006afa0 0x7f67f006d460 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f680c16d910 tx=0x7f67fc00a000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.958 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.922+0000 7ffa70ff9700 1 -- 172.21.15.91:0/1082349052 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffa64054f60 con 0x7ffa74160b00 2023-11-24T06:08:07.958 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.924+0000 7f681472e700 1 -- 172.21.15.91:0/555313099 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f40052e0 con 0x7f67f8004850 2023-11-24T06:08:07.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.930+0000 7f682e4e8700 1 Processor -- start 2023-11-24T06:08:07.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.930+0000 7f682e4e8700 1 -- start start 2023-11-24T06:08:07.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.931+0000 7f6802ffd700 1 -- 172.21.15.91:0/555313099 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6808054510 con 0x7f67f8004850 2023-11-24T06:08:07.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.931+0000 7f682e4e8700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f68280ceff0 0x7f68280c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.931+0000 7f682e4e8700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68280c9690 con 0x7f68280c9210 2023-11-24T06:08:07.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.931+0000 7f682e4e8700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68280718e0 con 0x7f68280cdc60 2023-11-24T06:08:07.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.931+0000 7f682e4e8700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68280cf3d0 con 0x7f68280ceff0 2023-11-24T06:08:07.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.931+0000 7f6827fff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f68280ceff0 0x7f68280c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.931+0000 7f6827fff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f68280ceff0 0x7f68280c8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58056/0 (socket says 172.21.15.91:58056) 2023-11-24T06:08:07.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.931+0000 7f6827fff700 1 -- 172.21.15.91:0/3512754026 learned_addr learned my addr 172.21.15.91:0/3512754026 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.931+0000 7f6827fff700 1 -- 172.21.15.91:0/3512754026 >> v1:172.21.15.94:6789/0 conn(0x7f68280cdc60 legacy=0x7f6828161a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.931+0000 7f6827fff700 1 -- 172.21.15.91:0/3512754026 >> v1:172.21.15.91:6789/0 conn(0x7f68280c9210 legacy=0x7f68280cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:07.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.931+0000 7f6827fff700 1 -- 172.21.15.91:0/3512754026 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682815e4a0 con 0x7f68280ceff0 2023-11-24T06:08:07.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.932+0000 7f840effd700 1 -- 172.21.15.91:0/218484919 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8400004230 con 0x7f84100c93e0 2023-11-24T06:08:07.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.932+0000 7f840effd700 1 -- 172.21.15.91:0/218484919 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8400004390 con 0x7f84100c93e0 2023-11-24T06:08:07.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.932+0000 7f840effd700 1 -- 172.21.15.91:0/218484919 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84000036a0 con 0x7f84100c93e0 2023-11-24T06:08:07.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.933+0000 7f8416277700 1 -- 172.21.15.91:0/218484919 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84100c93e0 msgr2=0x7f84100cd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.933+0000 7f8416277700 1 --2- 172.21.15.91:0/218484919 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84100c93e0 0x7f84100cd550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8400009870 tx=0x7f8400009b80 comp rx=0 tx=0).stop 2023-11-24T06:08:07.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.922+0000 7f4370788700 1 Processor -- start 2023-11-24T06:08:07.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.922+0000 7f4370788700 1 -- start start 2023-11-24T06:08:07.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.923+0000 7f4370788700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43680cbb80 0x7f43680b5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.923+0000 7f4370788700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f436814bf10 0x7f43682006b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.923+0000 7f4370788700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4368165f70 0x7f43681edce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.923+0000 7f4370788700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43681ebd90 con 0x7f436814bf10 2023-11-24T06:08:07.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.923+0000 7f4370788700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43681647c0 con 0x7f43680cbb80 2023-11-24T06:08:07.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.923+0000 7f4370788700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4368164aa0 con 0x7f4368165f70 2023-11-24T06:08:07.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.923+0000 7f436dd23700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f436814bf10 0x7f43682006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.923+0000 7f436dd23700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f436814bf10 0x7f43682006b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58048/0 (socket says 172.21.15.91:58048) 2023-11-24T06:08:07.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.923+0000 7f436dd23700 1 -- 172.21.15.91:0/308474345 learned_addr learned my addr 172.21.15.91:0/308474345 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.923+0000 7f436ed25700 1 --2- 172.21.15.91:0/308474345 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4368165f70 0x7f43681edce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.924+0000 7f436ed25700 1 -- 172.21.15.91:0/308474345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43680cbb80 msgr2=0x7f43680b5150 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:07.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.924+0000 7f436ed25700 1 --2- 172.21.15.91:0/308474345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43680cbb80 0x7f43680b5150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.924+0000 7f436ed25700 1 -- 172.21.15.91:0/308474345 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f436814bf10 msgr2=0x7f43682006b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.924+0000 7f436ed25700 1 --2- 172.21.15.91:0/308474345 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f436814bf10 0x7f43682006b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.924+0000 7f436ed25700 1 -- 172.21.15.91:0/308474345 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43580094a0 con 0x7f4368165f70 2023-11-24T06:08:07.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.924+0000 7f436ed25700 1 --2- 172.21.15.91:0/308474345 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4368165f70 0x7f43681edce0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f436000caa0 tx=0x7f436000cdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.924+0000 7f435effd700 1 -- 172.21.15.91:0/308474345 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4360003c00 con 0x7f4368165f70 2023-11-24T06:08:07.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.924+0000 7f435effd700 1 -- 172.21.15.91:0/308474345 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4360003d60 con 0x7f4368165f70 2023-11-24T06:08:07.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.924+0000 7f435effd700 1 -- 172.21.15.91:0/308474345 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436000aa20 con 0x7f4368165f70 2023-11-24T06:08:07.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.924+0000 7f4370788700 1 -- 172.21.15.91:0/308474345 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4368164d80 con 0x7f4368165f70 2023-11-24T06:08:07.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.924+0000 7f4370788700 1 -- 172.21.15.91:0/308474345 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43680b2690 con 0x7f4368165f70 2023-11-24T06:08:07.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.926+0000 7f4370788700 1 -- 172.21.15.91:0/308474345 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4368067f60 con 0x7f4368165f70 2023-11-24T06:08:07.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.927+0000 7f435effd700 1 -- 172.21.15.91:0/308474345 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f436000ab80 con 0x7f4368165f70 2023-11-24T06:08:07.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.928+0000 7f435effd700 1 --2- 172.21.15.91:0/308474345 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f434c06af20 0x7f434c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.964 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.928+0000 7f435effd700 1 -- 172.21.15.91:0/308474345 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4360089440 con 0x7f4368165f70 2023-11-24T06:08:07.964 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.930+0000 7f436e524700 1 --2- 172.21.15.91:0/308474345 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f434c06af20 0x7f434c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.964 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.932+0000 7f435effd700 1 -- 172.21.15.91:0/308474345 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4360054580 con 0x7f4368165f70 2023-11-24T06:08:07.964 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.932+0000 7f6827fff700 1 --2- 172.21.15.91:0/3512754026 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68280ceff0 0x7f68280c8cd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f681c0098f0 tx=0x7f681c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf264b70f9b006a3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.964 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.934+0000 7f6826ffd700 1 -- 172.21.15.91:0/3512754026 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c004230 con 0x7f68280ceff0 2023-11-24T06:08:07.964 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.934+0000 7f6826ffd700 1 -- 172.21.15.91:0/3512754026 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681c004390 con 0x7f68280ceff0 2023-11-24T06:08:07.964 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.934+0000 7f6826ffd700 1 -- 172.21.15.91:0/3512754026 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c0036a0 con 0x7f68280ceff0 2023-11-24T06:08:07.965 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.940+0000 7f436e524700 1 --2- 172.21.15.91:0/308474345 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f434c06af20 0x7f434c06d3e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4364001600 tx=0x7f4364007ad0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.965 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.941+0000 7f8416277700 1 -- 172.21.15.91:0/218484919 shutdown_connections 2023-11-24T06:08:07.965 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.941+0000 7f8416277700 1 --2- 172.21.15.91:0/218484919 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84100c93e0 0x7f84100cd550 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.965 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.941+0000 7f8416277700 1 -- 172.21.15.91:0/218484919 >> 172.21.15.91:0/218484919 conn(0x7f84100cf980 msgr2=0x7f84100cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.942+0000 7f682e4e8700 1 -- 172.21.15.91:0/3512754026 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68280ceff0 msgr2=0x7f68280c8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.942+0000 7f682e4e8700 1 --2- 172.21.15.91:0/3512754026 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68280ceff0 0x7f68280c8cd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f681c0098f0 tx=0x7f681c009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:07.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.942+0000 7f682e4e8700 1 -- 172.21.15.91:0/3512754026 shutdown_connections 2023-11-24T06:08:07.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.942+0000 7f682e4e8700 1 --2- 172.21.15.91:0/3512754026 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68280ceff0 0x7f68280c8cd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.942+0000 7f682e4e8700 1 -- 172.21.15.91:0/3512754026 >> 172.21.15.91:0/3512754026 conn(0x7f68280cf980 msgr2=0x7f68280cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:07.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.942+0000 7f682e4e8700 1 -- 172.21.15.91:0/3512754026 shutdown_connections 2023-11-24T06:08:07.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.942+0000 7f682e4e8700 1 -- 172.21.15.91:0/3512754026 wait complete. 2023-11-24T06:08:07.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.942+0000 7f8416277700 1 -- 172.21.15.91:0/218484919 shutdown_connections 2023-11-24T06:08:07.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.944+0000 7f682e4e8700 1 Processor -- start 2023-11-24T06:08:07.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.944+0000 7f682e4e8700 1 -- start start 2023-11-24T06:08:07.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.946+0000 7f682e4e8700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68280c9210 0x7f6828160690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.946+0000 7f682e4e8700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f68280cdc60 0x7f682813b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.946+0000 7f682e4e8700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68280ceff0 0x7f68281465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.946+0000 7f682e4e8700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68280b2740 con 0x7f68280ceff0 2023-11-24T06:08:07.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.946+0000 7f682e4e8700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68281614a0 con 0x7f68280c9210 2023-11-24T06:08:07.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.946+0000 7f682e4e8700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6828160ee0 con 0x7f68280cdc60 2023-11-24T06:08:07.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.946+0000 7f682ca85700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68280ceff0 0x7f68281465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.946+0000 7f682ca85700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68280ceff0 0x7f68281465d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58066/0 (socket says 172.21.15.91:58066) 2023-11-24T06:08:07.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.946+0000 7f682ca85700 1 -- 172.21.15.91:0/3356060716 learned_addr learned my addr 172.21.15.91:0/3356060716 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.946+0000 7f68277fe700 1 --2- 172.21.15.91:0/3356060716 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f68280cdc60 0x7f682813b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.946+0000 7f68277fe700 1 -- 172.21.15.91:0/3356060716 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68280c9210 msgr2=0x7f6828160690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:07.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.946+0000 7f68277fe700 1 --2- 172.21.15.91:0/3356060716 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68280c9210 0x7f6828160690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.946+0000 7f68277fe700 1 -- 172.21.15.91:0/3356060716 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68280ceff0 msgr2=0x7f68281465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:07.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.946+0000 7f68277fe700 1 --2- 172.21.15.91:0/3356060716 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68280ceff0 0x7f68281465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.946+0000 7f68277fe700 1 -- 172.21.15.91:0/3356060716 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c009550 con 0x7f68280cdc60 2023-11-24T06:08:07.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.947+0000 7f68277fe700 1 --2- 172.21.15.91:0/3356060716 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f68280cdc60 0x7f682813b360 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f682000be20 tx=0x7f6820009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.950+0000 7f6824ff9700 1 -- 172.21.15.91:0/3356060716 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682000cd10 con 0x7f68280cdc60 2023-11-24T06:08:07.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.950+0000 7f6824ff9700 1 -- 172.21.15.91:0/3356060716 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f682000ce70 con 0x7f68280cdc60 2023-11-24T06:08:07.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.950+0000 7f6824ff9700 1 -- 172.21.15.91:0/3356060716 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6820014660 con 0x7f68280cdc60 2023-11-24T06:08:07.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.950+0000 7f682e4e8700 1 -- 172.21.15.91:0/3356060716 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68281611c0 con 0x7f68280cdc60 2023-11-24T06:08:07.970 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.950+0000 7f682e4e8700 1 -- 172.21.15.91:0/3356060716 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6828158ee0 con 0x7f68280cdc60 2023-11-24T06:08:07.970 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.950+0000 7f8416277700 1 -- 172.21.15.91:0/218484919 wait complete. 2023-11-24T06:08:07.970 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.951+0000 7f8416277700 1 Processor -- start 2023-11-24T06:08:07.970 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.952+0000 7f8416277700 1 -- start start 2023-11-24T06:08:07.970 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.953+0000 7f8416277700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84100c93e0 0x7f8410145820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.970 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.953+0000 7f8416277700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84100cda90 0x7f841015ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.953+0000 7f8416277700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84100ceff0 0x7f8410160290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.953+0000 7f8416277700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8410145d60 con 0x7f84100c93e0 2023-11-24T06:08:07.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.953+0000 7f8416277700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841015b160 con 0x7f84100cda90 2023-11-24T06:08:07.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.953+0000 7f8416277700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f841006cf00 con 0x7f84100ceff0 2023-11-24T06:08:07.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.953+0000 7f840ffff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84100c93e0 0x7f8410145820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.953+0000 7f840ffff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84100c93e0 0x7f8410145820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58068/0 (socket says 172.21.15.91:58068) 2023-11-24T06:08:07.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.953+0000 7f840ffff700 1 -- 172.21.15.91:0/3946482516 learned_addr learned my addr 172.21.15.91:0/3946482516 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:07.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.954+0000 7f840ffff700 1 -- 172.21.15.91:0/3946482516 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84100ceff0 msgr2=0x7f8410160290 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:07.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.954+0000 7f840ffff700 1 --2- 172.21.15.91:0/3946482516 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84100ceff0 0x7f8410160290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.954+0000 7f840ffff700 1 -- 172.21.15.91:0/3946482516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84100cda90 msgr2=0x7f841015ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:07.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.954+0000 7f840ffff700 1 --2- 172.21.15.91:0/3946482516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84100cda90 0x7f841015ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:07.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.954+0000 7f840ffff700 1 -- 172.21.15.91:0/3946482516 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84000094a0 con 0x7f84100c93e0 2023-11-24T06:08:07.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.954+0000 7f840ffff700 1 --2- 172.21.15.91:0/3946482516 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84100c93e0 0x7f8410145820 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f840800cbc0 tx=0x7f840800cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.958+0000 7f6824ff9700 1 -- 172.21.15.91:0/3356060716 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6820006d50 con 0x7f68280cdc60 2023-11-24T06:08:07.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.958+0000 7f6824ff9700 1 --2- 172.21.15.91:0/3356060716 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f681006b0b0 0x7f681006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.958+0000 7f6824ff9700 1 -- 172.21.15.91:0/3356060716 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6820089190 con 0x7f68280cdc60 2023-11-24T06:08:07.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.959+0000 7f6827fff700 1 --2- 172.21.15.91:0/3356060716 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f681006b0b0 0x7f681006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.973 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.959+0000 7f6827fff700 1 --2- 172.21.15.91:0/3356060716 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f681006b0b0 0x7f681006d570 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f681c009c40 tx=0x7f681c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.973 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.966+0000 7f682e4e8700 1 -- 172.21.15.91:0/3356060716 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68080052e0 con 0x7f68280cdc60 2023-11-24T06:08:07.973 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.966+0000 7f840cff9700 1 -- 172.21.15.91:0/3946482516 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840800aaf0 con 0x7f84100c93e0 2023-11-24T06:08:07.973 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.966+0000 7f840cff9700 1 -- 172.21.15.91:0/3946482516 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f840800ac50 con 0x7f84100c93e0 2023-11-24T06:08:07.973 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.966+0000 7f840cff9700 1 -- 172.21.15.91:0/3946482516 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408012400 con 0x7f84100c93e0 2023-11-24T06:08:07.973 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.966+0000 7f8416277700 1 -- 172.21.15.91:0/3946482516 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f841006d1e0 con 0x7f84100c93e0 2023-11-24T06:08:07.973 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.966+0000 7f8416277700 1 -- 172.21.15.91:0/3946482516 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f841006d7a0 con 0x7f84100c93e0 2023-11-24T06:08:07.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.968+0000 7f8416277700 1 -- 172.21.15.91:0/3946482516 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f00052e0 con 0x7f84100c93e0 2023-11-24T06:08:07.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.970+0000 7f840cff9700 1 -- 172.21.15.91:0/3946482516 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f840801b3b0 con 0x7f84100c93e0 2023-11-24T06:08:07.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.970+0000 7f840cff9700 1 --2- 172.21.15.91:0/3946482516 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f83fc06afa0 0x7f83fc06d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:07.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.970+0000 7f840cff9700 1 -- 172.21.15.91:0/3946482516 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8408089700 con 0x7f84100c93e0 2023-11-24T06:08:07.975 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.971+0000 7f840f7fe700 1 --2- 172.21.15.91:0/3946482516 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f83fc06afa0 0x7f83fc06d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:07.975 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.971+0000 7f6824ff9700 1 -- 172.21.15.91:0/3356060716 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68200543b0 con 0x7f68280cdc60 2023-11-24T06:08:07.975 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.971+0000 7f840f7fe700 1 --2- 172.21.15.91:0/3946482516 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f83fc06afa0 0x7f83fc06d460 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f84100d0870 tx=0x7f8400013040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:07.989 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:07.978+0000 7f840cff9700 1 -- 172.21.15.91:0/3946482516 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f84080548d0 con 0x7f84100c93e0 2023-11-24T06:08:08.294 INFO:teuthology.orchestra.run.smithi091.stdout:55834574852 2023-11-24T06:08:08.298 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.285+0000 7f11a08f2700 1 -- 172.21.15.91:0/3238244695 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f119c0ca0a0 con 0x7f119c15db10 2023-11-24T06:08:08.298 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.288+0000 7f118affd700 1 -- 172.21.15.91:0/3238244695 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1190057d00 con 0x7f119c15db10 2023-11-24T06:08:08.302 INFO:teuthology.orchestra.run.smithi091.stdout:55834574854 2023-11-24T06:08:08.305 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.298+0000 7f0823088700 1 -- 172.21.15.91:0/3299192464 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f0808005860 con 0x7f0804004850 2023-11-24T06:08:08.305 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.298+0000 7f08197fa700 1 -- 172.21.15.91:0/3299192464 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0810023020 con 0x7f0804004850 2023-11-24T06:08:08.306 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.302+0000 7f11a08f2700 1 -- 172.21.15.91:0/3238244695 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f117c06af20 msgr2=0x7f117c06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:08.306 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.302+0000 7f11a08f2700 1 --2- 172.21.15.91:0/3238244695 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f117c06af20 0x7f117c06d3e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f119c141240 tx=0x7f118c008030 comp rx=0 tx=0).stop 2023-11-24T06:08:08.306 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.302+0000 7f11a08f2700 1 -- 172.21.15.91:0/3238244695 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f119c15db10 msgr2=0x7f119c200110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:08.306 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.302+0000 7f11a08f2700 1 --2- 172.21.15.91:0/3238244695 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f119c15db10 0x7f119c200110 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f119000b550 tx=0x7f119000b910 comp rx=0 tx=0).stop 2023-11-24T06:08:08.306 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.302+0000 7f11a08f2700 1 -- 172.21.15.91:0/3238244695 shutdown_connections 2023-11-24T06:08:08.306 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.302+0000 7f11a08f2700 1 --2- 172.21.15.91:0/3238244695 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f117c06af20 0x7f117c06d3e0 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f119c141240 tx=0x7f118c008030 comp rx=0 tx=0).stop 2023-11-24T06:08:08.307 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.302+0000 7f11a08f2700 1 --2- 172.21.15.91:0/3238244695 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f119c071390 0x7f119c1e3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.307 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.302+0000 7f11a08f2700 1 --2- 172.21.15.91:0/3238244695 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f119c15db10 0x7f119c200110 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.307 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.302+0000 7f11a08f2700 1 --2- 172.21.15.91:0/3238244695 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f119c0ca2c0 0x7f119c1e6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.307 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.302+0000 7f11a08f2700 1 -- 172.21.15.91:0/3238244695 >> 172.21.15.91:0/3238244695 conn(0x7f119c0cf980 msgr2=0x7f119c06d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:08.308 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.305+0000 7f11a08f2700 1 -- 172.21.15.91:0/3238244695 shutdown_connections 2023-11-24T06:08:08.309 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.306+0000 7f11a08f2700 1 -- 172.21.15.91:0/3238244695 wait complete. 2023-11-24T06:08:08.315 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.311+0000 7fe395f4e700 1 -- 172.21.15.91:0/2673177771 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe390068670 con 0x7fe39006e520 2023-11-24T06:08:08.316 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.312+0000 7f0823088700 1 -- 172.21.15.91:0/3299192464 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f080406cbc0 msgr2=0x7f080406f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:08.316 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.312+0000 7f0823088700 1 --2- 172.21.15.91:0/3299192464 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f080406cbc0 0x7f080406f080 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f080c015040 tx=0x7f080c007000 comp rx=0 tx=0).stop 2023-11-24T06:08:08.316 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.312+0000 7f0823088700 1 -- 172.21.15.91:0/3299192464 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0804004850 msgr2=0x7f081c156d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:08.316 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.312+0000 7f0823088700 1 --2- 172.21.15.91:0/3299192464 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0804004850 0x7f081c156d40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f081000e5d0 tx=0x7f081000e8e0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.317 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.312+0000 7f0823088700 1 -- 172.21.15.91:0/3299192464 shutdown_connections 2023-11-24T06:08:08.317 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.312+0000 7f0823088700 1 --2- 172.21.15.91:0/3299192464 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f080406cbc0 0x7f080406f080 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.317 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.312+0000 7f0823088700 1 --2- 172.21.15.91:0/3299192464 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f081c159290 0x7f081c13c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.317 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.312+0000 7f0823088700 1 --2- 172.21.15.91:0/3299192464 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0804004850 0x7f081c156d40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.317 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.312+0000 7f0823088700 1 --2- 172.21.15.91:0/3299192464 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f081c159800 0x7f081c156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.317 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.312+0000 7f0823088700 1 -- 172.21.15.91:0/3299192464 >> 172.21.15.91:0/3299192464 conn(0x7f081c0cf980 msgr2=0x7f081c143140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:08.317 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.312+0000 7fe377fff700 1 -- 172.21.15.91:0/2673177771 <== mon.1 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe38001b440 con 0x7fe39006e520 2023-11-24T06:08:08.318 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.312+0000 7f0823088700 1 -- 172.21.15.91:0/3299192464 shutdown_connections 2023-11-24T06:08:08.318 INFO:teuthology.orchestra.run.smithi091.stdout:55834574854 2023-11-24T06:08:08.320 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.316+0000 7f0823088700 1 -- 172.21.15.91:0/3299192464 wait complete. 2023-11-24T06:08:08.320 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.316+0000 7fe395f4e700 1 -- 172.21.15.91:0/2673177771 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe37c06afa0 msgr2=0x7fe37c06d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:08.320 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.316+0000 7fe395f4e700 1 --2- 172.21.15.91:0/2673177771 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe37c06afa0 0x7fe37c06d460 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe384009770 tx=0x7fe384006cd0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.320 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.316+0000 7fe395f4e700 1 -- 172.21.15.91:0/2673177771 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe39006e520 msgr2=0x7fe390138b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:08.320 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.316+0000 7fe395f4e700 1 --2- 172.21.15.91:0/2673177771 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe39006e520 0x7fe390138b40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe380000c00 tx=0x7fe380004750 comp rx=0 tx=0).stop 2023-11-24T06:08:08.321 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.317+0000 7fe395f4e700 1 -- 172.21.15.91:0/2673177771 shutdown_connections 2023-11-24T06:08:08.321 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.317+0000 7fe395f4e700 1 --2- 172.21.15.91:0/2673177771 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe37c06afa0 0x7fe37c06d460 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.321 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.317+0000 7fe395f4e700 1 --2- 172.21.15.91:0/2673177771 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe39006e520 0x7fe390138b40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.321 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.317+0000 7fe395f4e700 1 --2- 172.21.15.91:0/2673177771 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe39014c040 0x7fe39013c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.322 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.317+0000 7fe395f4e700 1 --2- 172.21.15.91:0/2673177771 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe39006e9a0 0x7fe390135290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.322 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.317+0000 7fe395f4e700 1 -- 172.21.15.91:0/2673177771 >> 172.21.15.91:0/2673177771 conn(0x7fe3900d0bc0 msgr2=0x7fe3900b1050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:08.325 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.322+0000 7fe395f4e700 1 -- 172.21.15.91:0/2673177771 shutdown_connections 2023-11-24T06:08:08.326 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.322+0000 7fe395f4e700 1 -- 172.21.15.91:0/2673177771 wait complete. 2023-11-24T06:08:08.328 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.0 2023-11-24T06:08:08.333 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.326+0000 7ffa7a7fb700 1 -- 172.21.15.91:0/1082349052 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ffa54005590 con 0x7ffa74160b00 2023-11-24T06:08:08.334 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.330+0000 7ffa70ff9700 1 -- 172.21.15.91:0/1082349052 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffa64028030 con 0x7ffa74160b00 2023-11-24T06:08:08.335 INFO:teuthology.orchestra.run.smithi091.stdout:55834574853 2023-11-24T06:08:08.337 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.2 2023-11-24T06:08:08.338 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.334+0000 7ffa7a7fb700 1 -- 172.21.15.91:0/1082349052 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ffa6006afa0 msgr2=0x7ffa6006d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:08.338 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.334+0000 7ffa7a7fb700 1 --2- 172.21.15.91:0/1082349052 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ffa6006afa0 0x7ffa6006d460 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c001d10 tx=0x7ffa6c009040 comp rx=0 tx=0).stop 2023-11-24T06:08:08.338 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.334+0000 7ffa7a7fb700 1 -- 172.21.15.91:0/1082349052 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa74160b00 msgr2=0x7ffa74162890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:08.338 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.334+0000 7ffa7a7fb700 1 --2- 172.21.15.91:0/1082349052 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa74160b00 0x7ffa74162890 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c005830 tx=0x7ffa64004860 comp rx=0 tx=0).stop 2023-11-24T06:08:08.339 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.334+0000 7ffa7a7fb700 1 -- 172.21.15.91:0/1082349052 shutdown_connections 2023-11-24T06:08:08.339 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.334+0000 7ffa7a7fb700 1 --2- 172.21.15.91:0/1082349052 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ffa6006afa0 0x7ffa6006d460 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c001d10 tx=0x7ffa6c009040 comp rx=0 tx=0).stop 2023-11-24T06:08:08.339 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.334+0000 7ffa7a7fb700 1 --2- 172.21.15.91:0/1082349052 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffa7414a880 0x7ffa7414ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.339 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.334+0000 7ffa7a7fb700 1 --2- 172.21.15.91:0/1082349052 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ffa5c004850 0x7ffa741457b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.339 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.334+0000 7ffa7a7fb700 1 --2- 172.21.15.91:0/1082349052 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffa74160b00 0x7ffa74162890 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.339 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.334+0000 7ffa7a7fb700 1 -- 172.21.15.91:0/1082349052 >> 172.21.15.91:0/1082349052 conn(0x7ffa740cf980 msgr2=0x7ffa74071000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:08.339 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.335+0000 7ffa7a7fb700 1 -- 172.21.15.91:0/1082349052 shutdown_connections 2023-11-24T06:08:08.339 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.336+0000 7ffa7a7fb700 1 -- 172.21.15.91:0/1082349052 wait complete. 2023-11-24T06:08:08.340 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.337+0000 7f681472e700 1 -- 172.21.15.91:0/555313099 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f67f4005590 con 0x7f67f8004850 2023-11-24T06:08:08.341 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.337+0000 7f6802ffd700 1 -- 172.21.15.91:0/555313099 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6808021030 con 0x7f67f8004850 2023-11-24T06:08:08.341 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.337+0000 7f682e4e8700 1 -- 172.21.15.91:0/3356060716 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6808005860 con 0x7f68280cdc60 2023-11-24T06:08:08.341 INFO:teuthology.orchestra.run.smithi091.stdout:55834574853 2023-11-24T06:08:08.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.339+0000 7f681472e700 1 -- 172.21.15.91:0/555313099 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f67f006afa0 msgr2=0x7f67f006d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:08.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.339+0000 7f681472e700 1 --2- 172.21.15.91:0/555313099 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f67f006afa0 0x7f67f006d460 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f680c16d910 tx=0x7f67fc00a000 comp rx=0 tx=0).stop 2023-11-24T06:08:08.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.339+0000 7f681472e700 1 -- 172.21.15.91:0/555313099 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f67f8004850 msgr2=0x7f680c16cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:08.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.339+0000 7f681472e700 1 --2- 172.21.15.91:0/555313099 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f67f8004850 0x7f680c16cf00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f680800e7c0 tx=0x7f680800eb80 comp rx=0 tx=0).stop 2023-11-24T06:08:08.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.340+0000 7f681472e700 1 -- 172.21.15.91:0/555313099 shutdown_connections 2023-11-24T06:08:08.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.340+0000 7f681472e700 1 --2- 172.21.15.91:0/555313099 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f67f006afa0 0x7f67f006d460 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.345 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.340+0000 7f681472e700 1 --2- 172.21.15.91:0/555313099 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f680c15a460 0x7f680c15a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.345 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.340+0000 7f681472e700 1 --2- 172.21.15.91:0/555313099 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f67f8004850 0x7f680c16cf00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.345 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.340+0000 7f681472e700 1 --2- 172.21.15.91:0/555313099 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f680c15c4f0 0x7f680c15c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.345 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.340+0000 7f681472e700 1 -- 172.21.15.91:0/555313099 >> 172.21.15.91:0/555313099 conn(0x7f680c0cf980 msgr2=0x7f680c06eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:08.345 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.340+0000 7f681472e700 1 -- 172.21.15.91:0/555313099 shutdown_connections 2023-11-24T06:08:08.345 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.340+0000 7f681472e700 1 -- 172.21.15.91:0/555313099 wait complete. 2023-11-24T06:08:08.345 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.340+0000 7f4370788700 1 -- 172.21.15.91:0/308474345 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f43680731b0 con 0x7f4368165f70 2023-11-24T06:08:08.345 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.340+0000 7f435effd700 1 -- 172.21.15.91:0/308474345 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4360057ba0 con 0x7f4368165f70 2023-11-24T06:08:08.346 INFO:teuthology.orchestra.run.smithi091.stdout:55834574853 2023-11-24T06:08:08.346 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.342+0000 7f6824ff9700 1 -- 172.21.15.91:0/3356060716 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6820019070 con 0x7f68280cdc60 2023-11-24T06:08:08.346 INFO:teuthology.orchestra.run.smithi091.stdout:55834574854 2023-11-24T06:08:08.348 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.1 2023-11-24T06:08:08.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.344+0000 7f4370788700 1 -- 172.21.15.91:0/308474345 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f434c06af20 msgr2=0x7f434c06d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:08.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.344+0000 7f4370788700 1 --2- 172.21.15.91:0/308474345 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f434c06af20 0x7f434c06d3e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4364001600 tx=0x7f4364007ad0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.344+0000 7f4370788700 1 -- 172.21.15.91:0/308474345 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4368165f70 msgr2=0x7f43681edce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:08.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.344+0000 7f4370788700 1 --2- 172.21.15.91:0/308474345 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4368165f70 0x7f43681edce0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f436000caa0 tx=0x7f436000cdb0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f4370788700 1 -- 172.21.15.91:0/308474345 shutdown_connections 2023-11-24T06:08:08.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f4370788700 1 --2- 172.21.15.91:0/308474345 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f434c06af20 0x7f434c06d3e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f4370788700 1 --2- 172.21.15.91:0/308474345 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f43680cbb80 0x7f43680b5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f4370788700 1 --2- 172.21.15.91:0/308474345 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4368165f70 0x7f43681edce0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f4370788700 1 --2- 172.21.15.91:0/308474345 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f436814bf10 0x7f43682006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.350 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f4370788700 1 -- 172.21.15.91:0/308474345 >> 172.21.15.91:0/308474345 conn(0x7f43680cf980 msgr2=0x7f43680b01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:08.351 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f682e4e8700 1 -- 172.21.15.91:0/3356060716 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f681006b0b0 msgr2=0x7f681006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:08.351 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f682e4e8700 1 --2- 172.21.15.91:0/3356060716 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f681006b0b0 0x7f681006d570 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f681c009c40 tx=0x7f681c008030 comp rx=0 tx=0).stop 2023-11-24T06:08:08.351 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f682e4e8700 1 -- 172.21.15.91:0/3356060716 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f68280cdc60 msgr2=0x7f682813b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:08.351 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f682e4e8700 1 --2- 172.21.15.91:0/3356060716 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f68280cdc60 0x7f682813b360 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f682000be20 tx=0x7f6820009f70 comp rx=0 tx=0).stop 2023-11-24T06:08:08.351 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f682e4e8700 1 -- 172.21.15.91:0/3356060716 shutdown_connections 2023-11-24T06:08:08.351 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f682e4e8700 1 --2- 172.21.15.91:0/3356060716 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f681006b0b0 0x7f681006d570 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.351 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f682e4e8700 1 --2- 172.21.15.91:0/3356060716 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f68280c9210 0x7f6828160690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.352 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f682e4e8700 1 --2- 172.21.15.91:0/3356060716 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68280ceff0 0x7f68281465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.352 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f682e4e8700 1 --2- 172.21.15.91:0/3356060716 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f68280cdc60 0x7f682813b360 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.352 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f682e4e8700 1 -- 172.21.15.91:0/3356060716 >> 172.21.15.91:0/3356060716 conn(0x7f68280cf980 msgr2=0x7f682815c030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:08.352 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f682e4e8700 1 -- 172.21.15.91:0/3356060716 shutdown_connections 2023-11-24T06:08:08.352 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.345+0000 7f682e4e8700 1 -- 172.21.15.91:0/3356060716 wait complete. 2023-11-24T06:08:08.352 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.346+0000 7f4370788700 1 -- 172.21.15.91:0/308474345 shutdown_connections 2023-11-24T06:08:08.352 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.346+0000 7f4370788700 1 -- 172.21.15.91:0/308474345 wait complete. 2023-11-24T06:08:08.357 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.4 2023-11-24T06:08:08.364 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.360+0000 7f8416277700 1 -- 172.21.15.91:0/3946482516 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f83f0005860 con 0x7f84100c93e0 2023-11-24T06:08:08.364 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.361+0000 7f840cff9700 1 -- 172.21.15.91:0/3946482516 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8408021020 con 0x7f84100c93e0 2023-11-24T06:08:08.365 INFO:teuthology.orchestra.run.smithi091.stdout:55834574853 2023-11-24T06:08:08.365 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.7 2023-11-24T06:08:08.366 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.6 2023-11-24T06:08:08.367 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.363+0000 7f8416277700 1 -- 172.21.15.91:0/3946482516 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f83fc06afa0 msgr2=0x7f83fc06d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:08.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.363+0000 7f8416277700 1 --2- 172.21.15.91:0/3946482516 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f83fc06afa0 0x7f83fc06d460 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f84100d0870 tx=0x7f8400013040 comp rx=0 tx=0).stop 2023-11-24T06:08:08.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.363+0000 7f8416277700 1 -- 172.21.15.91:0/3946482516 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84100c93e0 msgr2=0x7f8410145820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:08.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.363+0000 7f8416277700 1 --2- 172.21.15.91:0/3946482516 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84100c93e0 0x7f8410145820 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f840800cbc0 tx=0x7f840800cf80 comp rx=0 tx=0).stop 2023-11-24T06:08:08.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.363+0000 7f8416277700 1 -- 172.21.15.91:0/3946482516 shutdown_connections 2023-11-24T06:08:08.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.363+0000 7f8416277700 1 --2- 172.21.15.91:0/3946482516 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f83fc06afa0 0x7f83fc06d460 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.363+0000 7f8416277700 1 --2- 172.21.15.91:0/3946482516 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84100ceff0 0x7f8410160290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.363+0000 7f8416277700 1 --2- 172.21.15.91:0/3946482516 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84100c93e0 0x7f8410145820 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.363+0000 7f8416277700 1 --2- 172.21.15.91:0/3946482516 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84100cda90 0x7f841015ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:08.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.363+0000 7f8416277700 1 -- 172.21.15.91:0/3946482516 >> 172.21.15.91:0/3946482516 conn(0x7f84100cf980 msgr2=0x7f84100725e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:08.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.363+0000 7f8416277700 1 -- 172.21.15.91:0/3946482516 shutdown_connections 2023-11-24T06:08:08.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:08.363+0000 7f8416277700 1 -- 172.21.15.91:0/3946482516 wait complete. 2023-11-24T06:08:08.369 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.5 2023-11-24T06:08:08.379 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.3 2023-11-24T06:08:09.329 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-24T06:08:09.338 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-24T06:08:09.349 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-24T06:08:09.357 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-11-24T06:08:09.366 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-11-24T06:08:09.366 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-11-24T06:08:09.370 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-11-24T06:08:09.381 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-11-24T06:08:09.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.540+0000 7feb7c37b700 1 Processor -- start 2023-11-24T06:08:09.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.540+0000 7feb7c37b700 1 -- start start 2023-11-24T06:08:09.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.541+0000 7feb7c37b700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7feb7415db10 0x7feb7415def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.541+0000 7feb7c37b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7feb740ca2c0 0x7feb740ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.541+0000 7feb7c37b700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb740ce000 con 0x7feb74071390 2023-11-24T06:08:09.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.541+0000 7feb7c37b700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb740ce2c0 con 0x7feb7415db10 2023-11-24T06:08:09.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.541+0000 7feb7c37b700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb740ce5a0 con 0x7feb740ca2c0 2023-11-24T06:08:09.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.542+0000 7feb7a117700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7feb7415db10 0x7feb7415def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.542+0000 7feb7a117700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7feb7415db10 0x7feb7415def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58078/0 (socket says 172.21.15.91:58078) 2023-11-24T06:08:09.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.542+0000 7feb7a117700 1 -- 172.21.15.91:0/4120565349 learned_addr learned my addr 172.21.15.91:0/4120565349 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:09.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.542+0000 7feb7a117700 1 -- 172.21.15.91:0/4120565349 >> v2:172.21.15.94:3300/0 conn(0x7feb740ca2c0 msgr2=0x7feb740ca6d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T06:08:09.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.542+0000 7feb7a117700 1 --2- 172.21.15.91:0/4120565349 >> v2:172.21.15.94:3300/0 conn(0x7feb740ca2c0 0x7feb740ca6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.542+0000 7feb7a117700 1 -- 172.21.15.91:0/4120565349 >> v1:172.21.15.91:6789/0 conn(0x7feb74071390 legacy=0x7feb7415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:09.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.542+0000 7feb7a117700 1 -- 172.21.15.91:0/4120565349 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb740ce880 con 0x7feb7415db10 2023-11-24T06:08:09.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.543+0000 7feb7a117700 1 --2- 172.21.15.91:0/4120565349 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feb7415db10 0x7feb7415def0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7feb74070ea0 tx=0x7feb6400d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ecfb4907175431 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.543+0000 7feb79115700 1 -- 172.21.15.91:0/4120565349 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb64010070 con 0x7feb7415db10 2023-11-24T06:08:09.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.543+0000 7feb79115700 1 -- 172.21.15.91:0/4120565349 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6400da40 con 0x7feb7415db10 2023-11-24T06:08:09.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.543+0000 7feb79115700 1 -- 172.21.15.91:0/4120565349 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6400ddd0 con 0x7feb7415db10 2023-11-24T06:08:09.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.544+0000 7feb7c37b700 1 -- 172.21.15.91:0/4120565349 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feb7415db10 msgr2=0x7feb7415def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.544+0000 7feb7c37b700 1 --2- 172.21.15.91:0/4120565349 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feb7415db10 0x7feb7415def0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7feb74070ea0 tx=0x7feb6400d330 comp rx=0 tx=0).stop 2023-11-24T06:08:09.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.545+0000 7feb7c37b700 1 -- 172.21.15.91:0/4120565349 shutdown_connections 2023-11-24T06:08:09.550 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.545+0000 7feb7c37b700 1 --2- 172.21.15.91:0/4120565349 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feb7415db10 0x7feb7415def0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.550 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.545+0000 7feb7c37b700 1 --2- 172.21.15.91:0/4120565349 >> v2:172.21.15.94:3300/0 conn(0x7feb740ca2c0 0x7feb740ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.550 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.545+0000 7feb7c37b700 1 -- 172.21.15.91:0/4120565349 >> 172.21.15.91:0/4120565349 conn(0x7feb740cf980 msgr2=0x7feb740cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:09.550 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.546+0000 7feb7c37b700 1 -- 172.21.15.91:0/4120565349 shutdown_connections 2023-11-24T06:08:09.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.546+0000 7feb7c37b700 1 -- 172.21.15.91:0/4120565349 wait complete. 2023-11-24T06:08:09.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.547+0000 7feb7c37b700 1 Processor -- start 2023-11-24T06:08:09.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.547+0000 7feb7c37b700 1 -- start start 2023-11-24T06:08:09.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.548+0000 7feb7c37b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feb74071390 0x7feb740b0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.548+0000 7feb7c37b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb740ca2c0 0x7feb741387f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.548+0000 7feb7c37b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feb7415db10 0x7feb74138d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.548+0000 7feb7c37b700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb74154050 con 0x7feb7415db10 2023-11-24T06:08:09.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.548+0000 7feb7c37b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb741fa270 con 0x7feb740ca2c0 2023-11-24T06:08:09.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.548+0000 7feb7c37b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb74137390 con 0x7feb74071390 2023-11-24T06:08:09.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.549+0000 7feb7a918700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feb7415db10 0x7feb74138d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.549+0000 7feb7a918700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feb7415db10 0x7feb74138d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58086/0 (socket says 172.21.15.91:58086) 2023-11-24T06:08:09.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.549+0000 7feb7a918700 1 -- 172.21.15.91:0/3914335925 learned_addr learned my addr 172.21.15.91:0/3914335925 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:09.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.549+0000 7feb7a117700 1 --2- 172.21.15.91:0/3914335925 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feb74071390 0x7feb740b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.549+0000 7feb79916700 1 --2- 172.21.15.91:0/3914335925 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb740ca2c0 0x7feb741387f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.549+0000 7feb7a918700 1 -- 172.21.15.91:0/3914335925 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feb74071390 msgr2=0x7feb740b0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.549+0000 7feb7a918700 1 --2- 172.21.15.91:0/3914335925 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feb74071390 0x7feb740b0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.549+0000 7feb7a918700 1 -- 172.21.15.91:0/3914335925 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb740ca2c0 msgr2=0x7feb741387f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.549+0000 7feb7a918700 1 --2- 172.21.15.91:0/3914335925 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb740ca2c0 0x7feb741387f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.549+0000 7feb7a918700 1 -- 172.21.15.91:0/3914335925 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb640085d0 con 0x7feb7415db10 2023-11-24T06:08:09.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.550+0000 7feb7a918700 1 --2- 172.21.15.91:0/3914335925 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feb7415db10 0x7feb74138d30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00b990 tx=0x7feb6c00bca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.550+0000 7feb6affd700 1 -- 172.21.15.91:0/3914335925 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c00d890 con 0x7feb7415db10 2023-11-24T06:08:09.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.550+0000 7feb6affd700 1 -- 172.21.15.91:0/3914335925 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6c00d9f0 con 0x7feb7415db10 2023-11-24T06:08:09.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.550+0000 7feb6affd700 1 -- 172.21.15.91:0/3914335925 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c00dda0 con 0x7feb7415db10 2023-11-24T06:08:09.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.551+0000 7feb7c37b700 1 -- 172.21.15.91:0/3914335925 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb74136dd0 con 0x7feb7415db10 2023-11-24T06:08:09.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.551+0000 7feb7c37b700 1 -- 172.21.15.91:0/3914335925 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb740b0d80 con 0x7feb7415db10 2023-11-24T06:08:09.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.553+0000 7feb6affd700 1 -- 172.21.15.91:0/3914335925 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7feb6c006d50 con 0x7feb7415db10 2023-11-24T06:08:09.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.553+0000 7feb7c37b700 1 -- 172.21.15.91:0/3914335925 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb74067f60 con 0x7feb7415db10 2023-11-24T06:08:09.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.555+0000 7feb6affd700 1 --2- 172.21.15.91:0/3914335925 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7feb5806afa0 0x7feb5806d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.555+0000 7feb6affd700 1 -- 172.21.15.91:0/3914335925 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb6c089510 con 0x7feb7415db10 2023-11-24T06:08:09.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.560+0000 7feb7a117700 1 --2- 172.21.15.91:0/3914335925 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7feb5806afa0 0x7feb5806d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.561+0000 7feb7a117700 1 --2- 172.21.15.91:0/3914335925 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7feb5806afa0 0x7feb5806d460 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7feb6400d370 tx=0x7feb6400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.573 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.569+0000 7feb6affd700 1 -- 172.21.15.91:0/3914335925 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb6c054650 con 0x7feb7415db10 2023-11-24T06:08:09.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.581+0000 7fd577791700 1 Processor -- start 2023-11-24T06:08:09.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.582+0000 7fd577791700 1 -- start start 2023-11-24T06:08:09.587 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.583+0000 7fd577791700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd570145700 0x7fd57006e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.587 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.583+0000 7fd577791700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd57006eef0 0x7fd570070e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.587 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.583+0000 7fd577791700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5700cd550 con 0x7fd570071390 2023-11-24T06:08:09.587 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.583+0000 7fd577791700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd570071bc0 con 0x7fd57006eef0 2023-11-24T06:08:09.587 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.583+0000 7fd577791700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd570071ea0 con 0x7fd570145700 2023-11-24T06:08:09.588 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7fd574d2c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd57006eef0 0x7fd570070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.588 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7fd574d2c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd57006eef0 0x7fd570070e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58094/0 (socket says 172.21.15.91:58094) 2023-11-24T06:08:09.588 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7fd574d2c700 1 -- 172.21.15.91:0/3630350679 learned_addr learned my addr 172.21.15.91:0/3630350679 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:09.588 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7fd574d2c700 1 -- 172.21.15.91:0/3630350679 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd570145700 msgr2=0x7fd57006e9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T06:08:09.589 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7fd574d2c700 1 --2- 172.21.15.91:0/3630350679 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd570145700 0x7fd57006e9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.589 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7fd574d2c700 1 -- 172.21.15.91:0/3630350679 >> v1:172.21.15.91:6789/0 conn(0x7fd570071390 legacy=0x7fd57015c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.589 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7fd574d2c700 1 -- 172.21.15.91:0/3630350679 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd570072180 con 0x7fd57006eef0 2023-11-24T06:08:09.589 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7fd574d2c700 1 --2- 172.21.15.91:0/3630350679 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd57006eef0 0x7fd570070e50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd56c0098f0 tx=0x7fd56c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1840e3e637cf74 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.589 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7fd567fff700 1 -- 172.21.15.91:0/3630350679 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c00b8a0 con 0x7fd57006eef0 2023-11-24T06:08:09.589 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7fd567fff700 1 -- 172.21.15.91:0/3630350679 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56c00ba00 con 0x7fd57006eef0 2023-11-24T06:08:09.589 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7fd567fff700 1 -- 172.21.15.91:0/3630350679 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c00bd90 con 0x7fd57006eef0 2023-11-24T06:08:09.590 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.582+0000 7f9a62d9c700 1 Processor -- start 2023-11-24T06:08:09.590 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.582+0000 7f9a62d9c700 1 -- start start 2023-11-24T06:08:09.590 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.583+0000 7f9a62d9c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9a5c070a80 0x7f9a5c070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.590 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.583+0000 7f9a62d9c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9a5c166600 0x7f9a5c0cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.590 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.583+0000 7f9a62d9c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a5c0ca8e0 con 0x7f9a5c145870 2023-11-24T06:08:09.590 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.583+0000 7f9a62d9c700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a5c166a50 con 0x7f9a5c166600 2023-11-24T06:08:09.590 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7f9a62d9c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c0713a0 con 0x7f9a5c070a80 2023-11-24T06:08:09.590 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7f9a61339700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9a5c166600 0x7f9a5c0cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.590 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7f9a61339700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f9a5c166600 0x7f9a5c0cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:42494/0 (socket says 172.21.15.91:42494) 2023-11-24T06:08:09.590 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7f9a61339700 1 -- 172.21.15.91:0/1783935211 learned_addr learned my addr 172.21.15.91:0/1783935211 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:09.590 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7f9a61339700 1 -- 172.21.15.91:0/1783935211 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9a5c070a80 msgr2=0x7f9a5c070e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:09.591 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7f9a61339700 1 --2- 172.21.15.91:0/1783935211 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9a5c070a80 0x7f9a5c070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.591 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7f9a61339700 1 -- 172.21.15.91:0/1783935211 >> v1:172.21.15.94:6789/0 conn(0x7f9a5c145870 legacy=0x7f9a5c165ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:09.591 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.584+0000 7f9a61339700 1 -- 172.21.15.91:0/1783935211 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c0c9fd0 con 0x7f9a5c166600 2023-11-24T06:08:09.591 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.585+0000 7fd577791700 1 -- 172.21.15.91:0/3630350679 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd57006eef0 msgr2=0x7fd570070e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.591 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.585+0000 7fd577791700 1 --2- 172.21.15.91:0/3630350679 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd57006eef0 0x7fd570070e50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd56c0098f0 tx=0x7fd56c009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:09.591 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.586+0000 7fd577791700 1 -- 172.21.15.91:0/3630350679 shutdown_connections 2023-11-24T06:08:09.591 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.586+0000 7fd577791700 1 --2- 172.21.15.91:0/3630350679 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd570145700 0x7fd57006e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.591 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.586+0000 7fd577791700 1 --2- 172.21.15.91:0/3630350679 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd57006eef0 0x7fd570070e50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.591 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.586+0000 7fd577791700 1 -- 172.21.15.91:0/3630350679 >> 172.21.15.91:0/3630350679 conn(0x7fd5700cf980 msgr2=0x7fd5700cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:09.592 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.586+0000 7f9a61339700 1 --2- 172.21.15.91:0/1783935211 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c166600 0x7f9a5c0cad70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9a54009e50 tx=0x7f9a5400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b42b0728192fcce server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.592 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.586+0000 7ff09d2ce700 1 Processor -- start 2023-11-24T06:08:09.592 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.586+0000 7fd577791700 1 -- 172.21.15.91:0/3630350679 shutdown_connections 2023-11-24T06:08:09.593 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.586+0000 7ff09d2ce700 1 -- start start 2023-11-24T06:08:09.593 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.586+0000 7fd577791700 1 -- 172.21.15.91:0/3630350679 wait complete. 2023-11-24T06:08:09.593 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.587+0000 7ff09d2ce700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff09815abb0 0x7ff0980b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.593 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.587+0000 7ff09d2ce700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff0980724a0 0x7ff0980cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.593 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.587+0000 7ff09d2ce700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09806e550 con 0x7ff0980b4d50 2023-11-24T06:08:09.594 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.587+0000 7ff09d2ce700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0980b51d0 con 0x7ff09815abb0 2023-11-24T06:08:09.594 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.587+0000 7ff09d2ce700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff09806f3b0 con 0x7ff0980724a0 2023-11-24T06:08:09.594 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.587+0000 7fd577791700 1 Processor -- start 2023-11-24T06:08:09.594 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.587+0000 7fd577791700 1 -- start start 2023-11-24T06:08:09.594 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.588+0000 7f9a5b7fe700 1 -- 172.21.15.91:0/1783935211 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5400e070 con 0x7f9a5c166600 2023-11-24T06:08:09.594 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.588+0000 7f9a5b7fe700 1 -- 172.21.15.91:0/1783935211 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a54003a20 con 0x7f9a5c166600 2023-11-24T06:08:09.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.588+0000 7f9a5b7fe700 1 -- 172.21.15.91:0/1783935211 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a54003db0 con 0x7f9a5c166600 2023-11-24T06:08:09.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.588+0000 7ff096ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff09815abb0 0x7ff0980b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.588+0000 7ff096ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff09815abb0 0x7ff0980b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58108/0 (socket says 172.21.15.91:58108) 2023-11-24T06:08:09.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.588+0000 7ff096ffd700 1 -- 172.21.15.91:0/2644905952 learned_addr learned my addr 172.21.15.91:0/2644905952 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:09.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.588+0000 7ff0977fe700 1 --2- 172.21.15.91:0/2644905952 >> v2:172.21.15.94:3300/0 conn(0x7ff0980724a0 0x7ff0980cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.588+0000 7ff096ffd700 1 -- 172.21.15.91:0/2644905952 >> v2:172.21.15.94:3300/0 conn(0x7ff0980724a0 msgr2=0x7ff0980cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:09.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.588+0000 7ff096ffd700 1 --2- 172.21.15.91:0/2644905952 >> v2:172.21.15.94:3300/0 conn(0x7ff0980724a0 0x7ff0980cdee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.588+0000 7ff096ffd700 1 -- 172.21.15.91:0/2644905952 >> v1:172.21.15.94:6789/0 conn(0x7ff0980b4d50 legacy=0x7ff098071d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.588+0000 7ff096ffd700 1 -- 172.21.15.91:0/2644905952 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09806e7a0 con 0x7ff09815abb0 2023-11-24T06:08:09.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.588+0000 7ff096ffd700 1 --2- 172.21.15.91:0/2644905952 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff09815abb0 0x7ff0980b4810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff0980d0990 tx=0x7ff08c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba6856c34691eb69 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7f9a62d9c700 1 -- 172.21.15.91:0/1783935211 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c166600 msgr2=0x7f9a5c0cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7f9a62d9c700 1 --2- 172.21.15.91:0/1783935211 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c166600 0x7f9a5c0cad70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9a54009e50 tx=0x7f9a5400b330 comp rx=0 tx=0).stop 2023-11-24T06:08:09.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7fd577791700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd57006eef0 0x7fd5700cf140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7fd577791700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd570071390 0x7fd57016a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7fd577791700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd570145700 0x7fd570139f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7fd577791700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5701681b0 con 0x7fd570145700 2023-11-24T06:08:09.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7fd577791700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5700cf680 con 0x7fd570071390 2023-11-24T06:08:09.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7fd577791700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5701678a0 con 0x7fd57006eef0 2023-11-24T06:08:09.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7f9a62d9c700 1 -- 172.21.15.91:0/1783935211 shutdown_connections 2023-11-24T06:08:09.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7f9a62d9c700 1 --2- 172.21.15.91:0/1783935211 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c166600 0x7f9a5c0cad70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7f9a62d9c700 1 --2- 172.21.15.91:0/1783935211 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9a5c070a80 0x7f9a5c070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7f9a62d9c700 1 -- 172.21.15.91:0/1783935211 >> 172.21.15.91:0/1783935211 conn(0x7f9a5c0cf980 msgr2=0x7f9a5c0cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:09.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7fd57552d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd57006eef0 0x7fd5700cf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7fd57552d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd57006eef0 0x7fd5700cf140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:41542/0 (socket says 172.21.15.91:41542) 2023-11-24T06:08:09.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7fd57552d700 1 -- 172.21.15.91:0/3358550477 learned_addr learned my addr 172.21.15.91:0/3358550477 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:09.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7f9a62d9c700 1 -- 172.21.15.91:0/1783935211 shutdown_connections 2023-11-24T06:08:09.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7fd57552d700 1 -- 172.21.15.91:0/3358550477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd570071390 msgr2=0x7fd57016a640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:09.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7fd57552d700 1 --2- 172.21.15.91:0/3358550477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd570071390 0x7fd57016a640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7fd57552d700 1 -- 172.21.15.91:0/3358550477 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd570145700 msgr2=0x7fd570139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7fd57552d700 1 --2- 172.21.15.91:0/3358550477 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd570145700 0x7fd570139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7fd57552d700 1 -- 172.21.15.91:0/3358550477 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56c009550 con 0x7fd57006eef0 2023-11-24T06:08:09.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7fd57552d700 1 --2- 172.21.15.91:0/3358550477 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd57006eef0 0x7fd5700cf140 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd56000b470 tx=0x7fd56000b830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.600 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7ff095ffb700 1 -- 172.21.15.91:0/2644905952 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c00b870 con 0x7ff09815abb0 2023-11-24T06:08:09.600 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7f9a62d9c700 1 -- 172.21.15.91:0/1783935211 wait complete. 2023-11-24T06:08:09.600 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7ff095ffb700 1 -- 172.21.15.91:0/2644905952 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08c00b9d0 con 0x7ff09815abb0 2023-11-24T06:08:09.600 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.589+0000 7ff095ffb700 1 -- 172.21.15.91:0/2644905952 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c00bd40 con 0x7ff09815abb0 2023-11-24T06:08:09.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.590+0000 7fd565ffb700 1 -- 172.21.15.91:0/3358550477 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd560011830 con 0x7fd57006eef0 2023-11-24T06:08:09.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.590+0000 7fd577791700 1 -- 172.21.15.91:0/3358550477 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57016b0c0 con 0x7fd57006eef0 2023-11-24T06:08:09.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.590+0000 7f9a62d9c700 1 Processor -- start 2023-11-24T06:08:09.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.590+0000 7fd565ffb700 1 -- 172.21.15.91:0/3358550477 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd560011990 con 0x7fd57006eef0 2023-11-24T06:08:09.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.590+0000 7fd565ffb700 1 -- 172.21.15.91:0/3358550477 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd560011d40 con 0x7fd57006eef0 2023-11-24T06:08:09.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.590+0000 7fd577791700 1 -- 172.21.15.91:0/3358550477 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd570167dd0 con 0x7fd57006eef0 2023-11-24T06:08:09.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.590+0000 7ff09d2ce700 1 -- 172.21.15.91:0/2644905952 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff09815abb0 msgr2=0x7ff0980b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.590+0000 7ff09d2ce700 1 --2- 172.21.15.91:0/2644905952 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff09815abb0 0x7ff0980b4810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff0980d0990 tx=0x7ff08c009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.590+0000 7f9a62d9c700 1 -- start start 2023-11-24T06:08:09.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.591+0000 7ff09d2ce700 1 -- 172.21.15.91:0/2644905952 shutdown_connections 2023-11-24T06:08:09.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.591+0000 7ff09d2ce700 1 --2- 172.21.15.91:0/2644905952 >> v2:172.21.15.94:3300/0 conn(0x7ff0980724a0 0x7ff0980cdee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.591+0000 7ff09d2ce700 1 --2- 172.21.15.91:0/2644905952 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff09815abb0 0x7ff0980b4810 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.591+0000 7ff09d2ce700 1 -- 172.21.15.91:0/2644905952 >> 172.21.15.91:0/2644905952 conn(0x7ff0980d0aa0 msgr2=0x7ff0980d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:09.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7ff09d2ce700 1 -- 172.21.15.91:0/2644905952 shutdown_connections 2023-11-24T06:08:09.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f9a62d9c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9a5c070a80 0x7f9a5c143d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f9a62d9c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c145870 0x7f9a5c0b48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f9a62d9c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9a5c166600 0x7f9a5c13fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f9a62d9c700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c0b44b0 con 0x7f9a5c166600 2023-11-24T06:08:09.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f9a62d9c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c0b3830 con 0x7f9a5c145870 2023-11-24T06:08:09.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f9a62d9c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c0b3b10 con 0x7f9a5c070a80 2023-11-24T06:08:09.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f9a61339700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9a5c166600 0x7f9a5c13fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f9a5bfff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c145870 0x7f9a5c0b48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f9a60b38700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9a5c070a80 0x7f9a5c143d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f9a61339700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9a5c166600 0x7f9a5c13fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58120/0 (socket says 172.21.15.91:58120) 2023-11-24T06:08:09.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f9a61339700 1 -- 172.21.15.91:0/797809376 learned_addr learned my addr 172.21.15.91:0/797809376 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:09.605 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.591+0000 7f0a049ef700 1 Processor -- start 2023-11-24T06:08:09.605 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f0a049ef700 1 -- start start 2023-11-24T06:08:09.605 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7ff09d2ce700 1 -- 172.21.15.91:0/2644905952 wait complete. 2023-11-24T06:08:09.605 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f9a60b38700 1 -- 172.21.15.91:0/797809376 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c145870 msgr2=0x7f9a5c0b48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f9a60b38700 1 --2- 172.21.15.91:0/797809376 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c145870 0x7f9a5c0b48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f9a60b38700 1 -- 172.21.15.91:0/797809376 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9a5c166600 msgr2=0x7f9a5c13fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f9a60b38700 1 --2- 172.21.15.91:0/797809376 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9a5c166600 0x7f9a5c13fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.592+0000 7f9a60b38700 1 -- 172.21.15.91:0/797809376 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a54009b30 con 0x7f9a5c070a80 2023-11-24T06:08:09.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.593+0000 7f9a60b38700 1 --2- 172.21.15.91:0/797809376 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9a5c070a80 0x7f9a5c143d70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c00b450 tx=0x7f9a4c00b760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.593+0000 7f9a597fa700 1 -- 172.21.15.91:0/797809376 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c011830 con 0x7f9a5c070a80 2023-11-24T06:08:09.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.593+0000 7f9a597fa700 1 -- 172.21.15.91:0/797809376 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4c011990 con 0x7f9a5c070a80 2023-11-24T06:08:09.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.593+0000 7f9a597fa700 1 -- 172.21.15.91:0/797809376 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c011d40 con 0x7f9a5c070a80 2023-11-24T06:08:09.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.593+0000 7fd565ffb700 1 -- 172.21.15.91:0/3358550477 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd56001a400 con 0x7fd57006eef0 2023-11-24T06:08:09.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.594+0000 7ff09d2ce700 1 Processor -- start 2023-11-24T06:08:09.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.594+0000 7ff09d2ce700 1 -- start start 2023-11-24T06:08:09.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.593+0000 7f9a62d9c700 1 -- 172.21.15.91:0/797809376 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5c0b3df0 con 0x7f9a5c070a80 2023-11-24T06:08:09.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.593+0000 7f9a62d9c700 1 -- 172.21.15.91:0/797809376 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a5c1449f0 con 0x7f9a5c070a80 2023-11-24T06:08:09.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.593+0000 7f0a049ef700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f0a00070a80 0x7f0a00070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.593+0000 7f0a049ef700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a00166600 0x7f0a000cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.593+0000 7f0a049ef700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a00166a30 con 0x7f0a00145870 2023-11-24T06:08:09.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.593+0000 7f0a049ef700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a000ca8e0 con 0x7f0a00070a80 2023-11-24T06:08:09.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.593+0000 7f0a049ef700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000713a0 con 0x7f0a00166600 2023-11-24T06:08:09.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.594+0000 7f09fed9d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a00166600 0x7f0a000cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.594+0000 7f09fed9d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a00166600 0x7f0a000cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:41554/0 (socket says 172.21.15.91:41554) 2023-11-24T06:08:09.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.594+0000 7f09fed9d700 1 -- 172.21.15.91:0/1826310912 learned_addr learned my addr 172.21.15.91:0/1826310912 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:09.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.594+0000 7f09fed9d700 1 -- 172.21.15.91:0/1826310912 >> v2:172.21.15.91:3300/0 conn(0x7f0a00070a80 msgr2=0x7f0a00070e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-11-24T06:08:09.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.594+0000 7f09fed9d700 1 --2- 172.21.15.91:0/1826310912 >> v2:172.21.15.91:3300/0 conn(0x7f0a00070a80 0x7f0a00070e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.594+0000 7f09fed9d700 1 -- 172.21.15.91:0/1826310912 >> v1:172.21.15.91:6789/0 conn(0x7f0a00145870 legacy=0x7f0a00165ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:09.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.594+0000 7f09fed9d700 1 -- 172.21.15.91:0/1826310912 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a000c9fd0 con 0x7f0a00166600 2023-11-24T06:08:09.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.594+0000 7f09fed9d700 1 --2- 172.21.15.91:0/1826310912 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a00166600 0x7f0a000cad70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f09f8009e50 tx=0x7f09f800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48906c9de15590b4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.594+0000 7fd565ffb700 1 --2- 172.21.15.91:0/3358550477 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd55406af20 0x7fd55406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.594+0000 7fd565ffb700 1 -- 172.21.15.91:0/3358550477 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd560089470 con 0x7fd57006eef0 2023-11-24T06:08:09.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.594+0000 7fd574d2c700 1 --2- 172.21.15.91:0/3358550477 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd55406af20 0x7fd55406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.595+0000 7fd574d2c700 1 --2- 172.21.15.91:0/3358550477 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd55406af20 0x7fd55406d3e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd57013ad70 tx=0x7fd56c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.595+0000 7f09fd59a700 1 -- 172.21.15.91:0/1826310912 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f800e070 con 0x7f0a00166600 2023-11-24T06:08:09.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.595+0000 7f09fd59a700 1 -- 172.21.15.91:0/1826310912 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f8003a20 con 0x7f0a00166600 2023-11-24T06:08:09.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.595+0000 7f09fd59a700 1 -- 172.21.15.91:0/1826310912 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f8003dd0 con 0x7f0a00166600 2023-11-24T06:08:09.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.595+0000 7ff09d2ce700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0980724a0 0x7ff098149e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.595+0000 7ff09d2ce700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff0980b4d50 0x7ff09815c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.595+0000 7ff09d2ce700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff09815abb0 0x7ff0980ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.595+0000 7ff09d2ce700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09814b040 con 0x7ff0980b4d50 2023-11-24T06:08:09.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.595+0000 7ff09d2ce700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09814a350 con 0x7ff09815abb0 2023-11-24T06:08:09.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.595+0000 7ff09d2ce700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09815cd60 con 0x7ff0980724a0 2023-11-24T06:08:09.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.595+0000 7ff0967fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff0980b4d50 0x7ff09815c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.595+0000 7ff0967fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff0980b4d50 0x7ff09815c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58128/0 (socket says 172.21.15.91:58128) 2023-11-24T06:08:09.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.595+0000 7ff0967fc700 1 -- 172.21.15.91:0/3572858183 learned_addr learned my addr 172.21.15.91:0/3572858183 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:09.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.596+0000 7ff0967fc700 1 -- 172.21.15.91:0/3572858183 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0980724a0 msgr2=0x7ff098149e10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:09.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.596+0000 7ff0967fc700 1 --2- 172.21.15.91:0/3572858183 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0980724a0 0x7ff098149e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.596+0000 7ff0967fc700 1 -- 172.21.15.91:0/3572858183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff09815abb0 msgr2=0x7ff0980ceef0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:09.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.596+0000 7ff0967fc700 1 --2- 172.21.15.91:0/3572858183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff09815abb0 0x7ff0980ceef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.596+0000 7ff0967fc700 1 -- 172.21.15.91:0/3572858183 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c009550 con 0x7ff0980b4d50 2023-11-24T06:08:09.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.596+0000 7ff0967fc700 1 --2- 172.21.15.91:0/3572858183 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff0980b4d50 0x7ff09815c2a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff08800c4e0 tx=0x7ff08800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.596+0000 7f0a049ef700 1 -- 172.21.15.91:0/1826310912 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a00166600 msgr2=0x7f0a000cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.596+0000 7f0a049ef700 1 --2- 172.21.15.91:0/1826310912 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a00166600 0x7f0a000cad70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f09f8009e50 tx=0x7f09f800b330 comp rx=0 tx=0).stop 2023-11-24T06:08:09.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.596+0000 7f9a597fa700 1 -- 172.21.15.91:0/797809376 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9a4c01a400 con 0x7f9a5c070a80 2023-11-24T06:08:09.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.596+0000 7ff07f7fe700 1 -- 172.21.15.91:0/3572858183 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0880052d0 con 0x7ff0980b4d50 2023-11-24T06:08:09.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.596+0000 7ff07f7fe700 1 -- 172.21.15.91:0/3572858183 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff088005430 con 0x7ff0980b4d50 2023-11-24T06:08:09.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.596+0000 7ff07f7fe700 1 -- 172.21.15.91:0/3572858183 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0880038c0 con 0x7ff0980b4d50 2023-11-24T06:08:09.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.596+0000 7f9a62d9c700 1 -- 172.21.15.91:0/797809376 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a5c0b45f0 con 0x7f9a5c070a80 2023-11-24T06:08:09.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.597+0000 7f9a597fa700 1 --2- 172.21.15.91:0/797809376 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9a4806aed0 0x7f9a4806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.597+0000 7f9a597fa700 1 -- 172.21.15.91:0/797809376 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a4c0893d0 con 0x7f9a5c070a80 2023-11-24T06:08:09.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.597+0000 7ff09d2ce700 1 -- 172.21.15.91:0/3572858183 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff09815d040 con 0x7ff0980b4d50 2023-11-24T06:08:09.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.597+0000 7ff09d2ce700 1 -- 172.21.15.91:0/3572858183 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff09815d600 con 0x7ff0980b4d50 2023-11-24T06:08:09.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.597+0000 7f0a049ef700 1 -- 172.21.15.91:0/1826310912 shutdown_connections 2023-11-24T06:08:09.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.597+0000 7f0a049ef700 1 --2- 172.21.15.91:0/1826310912 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a00166600 0x7f0a000cad70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.597+0000 7f0a049ef700 1 --2- 172.21.15.91:0/1826310912 >> v2:172.21.15.91:3300/0 conn(0x7f0a00070a80 0x7f0a00070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.597+0000 7f0a049ef700 1 -- 172.21.15.91:0/1826310912 >> 172.21.15.91:0/1826310912 conn(0x7f0a000cf980 msgr2=0x7f0a000cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:09.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.597+0000 7f0a049ef700 1 -- 172.21.15.91:0/1826310912 shutdown_connections 2023-11-24T06:08:09.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.598+0000 7fd577791700 1 -- 172.21.15.91:0/3358550477 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5701682f0 con 0x7fd57006eef0 2023-11-24T06:08:09.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.598+0000 7f81ddff9700 1 Processor -- start 2023-11-24T06:08:09.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.598+0000 7f9a5bfff700 1 --2- 172.21.15.91:0/797809376 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9a4806aed0 0x7f9a4806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.598+0000 7f81ddff9700 1 -- start start 2023-11-24T06:08:09.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.599+0000 7f9a5bfff700 1 --2- 172.21.15.91:0/797809376 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9a4806aed0 0x7f9a4806d390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c144850 tx=0x7f9a50007ec0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.598+0000 7f0a049ef700 1 -- 172.21.15.91:0/1826310912 wait complete. 2023-11-24T06:08:09.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.599+0000 7f81ddff9700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f81d8071390 0x7f81d815a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.617 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.600+0000 7f81ddff9700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f81d80d04a0 0x7f81d815aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.617 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.600+0000 7f81ddff9700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81d815b3f0 con 0x7f81d815afa0 2023-11-24T06:08:09.617 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.600+0000 7f81ddff9700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d80ce000 con 0x7f81d8071390 2023-11-24T06:08:09.617 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.600+0000 7f81ddff9700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81d80cb2d0 con 0x7f81d80d04a0 2023-11-24T06:08:09.617 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.600+0000 7f0a049ef700 1 Processor -- start 2023-11-24T06:08:09.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.600+0000 7f81d77fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f81d8071390 0x7f81d815a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.600+0000 7f81d77fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f81d8071390 0x7f81d815a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58136/0 (socket says 172.21.15.91:58136) 2023-11-24T06:08:09.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.600+0000 7f81d77fe700 1 -- 172.21.15.91:0/586856665 learned_addr learned my addr 172.21.15.91:0/586856665 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:09.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.600+0000 7f81d6ffd700 1 --2- 172.21.15.91:0/586856665 >> v2:172.21.15.94:3300/0 conn(0x7f81d80d04a0 0x7f81d815aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.600+0000 7f81d77fe700 1 -- 172.21.15.91:0/586856665 >> v2:172.21.15.94:3300/0 conn(0x7f81d80d04a0 msgr2=0x7f81d815aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:09.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.600+0000 7f81d77fe700 1 --2- 172.21.15.91:0/586856665 >> v2:172.21.15.94:3300/0 conn(0x7f81d80d04a0 0x7f81d815aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.600+0000 7f81d77fe700 1 -- 172.21.15.91:0/586856665 >> v1:172.21.15.94:6789/0 conn(0x7f81d815afa0 legacy=0x7f81d815db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.600+0000 7f81d77fe700 1 -- 172.21.15.91:0/586856665 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d80ce2c0 con 0x7f81d8071390 2023-11-24T06:08:09.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.600+0000 7f81d77fe700 1 --2- 172.21.15.91:0/586856665 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f81d8071390 0x7f81d815a520 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f81c80098f0 tx=0x7f81c8009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45f449693a2f41b4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.599+0000 7ff09d2ce700 1 -- 172.21.15.91:0/3572858183 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0780052e0 con 0x7ff0980b4d50 2023-11-24T06:08:09.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.599+0000 7ff07f7fe700 1 -- 172.21.15.91:0/3572858183 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff088003a20 con 0x7ff0980b4d50 2023-11-24T06:08:09.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.600+0000 7f0a049ef700 1 -- start start 2023-11-24T06:08:09.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.601+0000 7f0a049ef700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a00070a80 0x7f0a0013a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.601+0000 7f0a049ef700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a00145870 0x7f0a000b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.601+0000 7f0a049ef700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0a00166600 0x7f0a000afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.601+0000 7f0a049ef700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000b4090 con 0x7f0a00166600 2023-11-24T06:08:09.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.601+0000 7ff03aead700 1 Processor -- start 2023-11-24T06:08:09.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.601+0000 7f0a049ef700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000b4370 con 0x7f0a00145870 2023-11-24T06:08:09.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.601+0000 7f0a049ef700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000b4650 con 0x7f0a00070a80 2023-11-24T06:08:09.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.601+0000 7ff03aead700 1 -- start start 2023-11-24T06:08:09.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.601+0000 7f09fdd9b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a00145870 0x7f0a000b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.601+0000 7f09fdd9b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a00145870 0x7f0a000b5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:42536/0 (socket says 172.21.15.91:42536) 2023-11-24T06:08:09.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.601+0000 7f09fdd9b700 1 -- 172.21.15.91:0/1623465741 learned_addr learned my addr 172.21.15.91:0/1623465741 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:09.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.601+0000 7f09fed9d700 1 --2- 172.21.15.91:0/1623465741 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0a00166600 0x7f0a000afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.602+0000 7f09fdd9b700 1 -- 172.21.15.91:0/1623465741 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a00070a80 msgr2=0x7f0a0013a840 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:09.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.602+0000 7f09fdd9b700 1 --2- 172.21.15.91:0/1623465741 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a00070a80 0x7f0a0013a840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.602+0000 7f09fdd9b700 1 -- 172.21.15.91:0/1623465741 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0a00166600 msgr2=0x7f0a000afd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.602+0000 7f09fdd9b700 1 --2- 172.21.15.91:0/1623465741 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0a00166600 0x7f0a000afd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.602+0000 7f09fdd9b700 1 -- 172.21.15.91:0/1623465741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f8009b30 con 0x7f0a00145870 2023-11-24T06:08:09.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.602+0000 7f09fdd9b700 1 --2- 172.21.15.91:0/1623465741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a00145870 0x7f0a000b5130 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f09f000b470 tx=0x7f09f000b780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.623 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.602+0000 7f09eeffd700 1 -- 172.21.15.91:0/1623465741 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f0011830 con 0x7f0a00145870 2023-11-24T06:08:09.623 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.602+0000 7f09eeffd700 1 -- 172.21.15.91:0/1623465741 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f00119b0 con 0x7f0a00145870 2023-11-24T06:08:09.623 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.602+0000 7f09eeffd700 1 -- 172.21.15.91:0/1623465741 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f0011d60 con 0x7f0a00145870 2023-11-24T06:08:09.623 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.602+0000 7f0a049ef700 1 -- 172.21.15.91:0/1623465741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a000b4930 con 0x7f0a00145870 2023-11-24T06:08:09.623 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.602+0000 7f0a049ef700 1 -- 172.21.15.91:0/1623465741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a00144fd0 con 0x7f0a00145870 2023-11-24T06:08:09.623 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.602+0000 7f9a597fa700 1 -- 172.21.15.91:0/797809376 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a5c0b45f0 con 0x7f9a5c070a80 2023-11-24T06:08:09.624 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.602+0000 7ff03aead700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff03406c5d0 0x7ff0340b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.624 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.602+0000 7ff03aead700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff0340b08f0 0x7ff0340d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.624 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.602+0000 7ff03aead700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0340b0d40 con 0x7ff03415afc0 2023-11-24T06:08:09.624 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.603+0000 7ff03944a700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff0340b08f0 0x7ff0340d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.624 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.603+0000 7ff03944a700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff0340b08f0 0x7ff0340d0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58150/0 (socket says 172.21.15.91:58150) 2023-11-24T06:08:09.624 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.603+0000 7ff03944a700 1 -- 172.21.15.91:0/511986322 learned_addr learned my addr 172.21.15.91:0/511986322 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:09.624 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.603+0000 7ff03aead700 1 -- 172.21.15.91:0/511986322 --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff03415f6b0 con 0x7ff0340b08f0 2023-11-24T06:08:09.624 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.603+0000 7ff03aead700 1 -- 172.21.15.91:0/511986322 --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff03415eb40 con 0x7ff03406c5d0 2023-11-24T06:08:09.625 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.603+0000 7ff07f7fe700 1 --2- 172.21.15.91:0/3572858183 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff08006b130 0x7ff08006d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.625 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.603+0000 7ff07f7fe700 1 -- 172.21.15.91:0/3572858183 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff088088a10 con 0x7ff0980b4d50 2023-11-24T06:08:09.625 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.604+0000 7ff03944a700 1 -- 172.21.15.91:0/511986322 >> v2:172.21.15.94:3300/0 conn(0x7ff03406c5d0 msgr2=0x7ff0340b03b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-11-24T06:08:09.625 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.604+0000 7ff03944a700 1 --2- 172.21.15.91:0/511986322 >> v2:172.21.15.94:3300/0 conn(0x7ff03406c5d0 0x7ff0340b03b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.625 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.604+0000 7ff03944a700 1 -- 172.21.15.91:0/511986322 >> v1:172.21.15.91:6789/0 conn(0x7ff03415afc0 legacy=0x7ff0340730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:09.625 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.604+0000 7ff03944a700 1 -- 172.21.15.91:0/511986322 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03415eda0 con 0x7ff0340b08f0 2023-11-24T06:08:09.625 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.604+0000 7ff03944a700 1 --2- 172.21.15.91:0/511986322 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff0340b08f0 0x7ff0340d0230 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff0340afe50 tx=0x7ff024009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d7f057fe6c3a35 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.626 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.604+0000 7f09eeffd700 1 -- 172.21.15.91:0/1623465741 <== mon.1 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f09f001a400 con 0x7f0a00145870 2023-11-24T06:08:09.626 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.604+0000 7f0a049ef700 1 -- 172.21.15.91:0/1623465741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a000b4e50 con 0x7f0a00145870 2023-11-24T06:08:09.626 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.604+0000 7ff096ffd700 1 --2- 172.21.15.91:0/3572858183 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff08006b130 0x7ff08006d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.627 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.604+0000 7f09eeffd700 1 --2- 172.21.15.91:0/1623465741 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f09e006aed0 0x7f09e006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.627 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.604+0000 7f09eeffd700 1 -- 172.21.15.91:0/1623465741 <== mon.1 v2:172.21.15.94:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09f00895b0 con 0x7f0a00145870 2023-11-24T06:08:09.627 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.605+0000 7f09fe59c700 1 --2- 172.21.15.91:0/1623465741 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f09e006aed0 0x7f09e006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.627 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.605+0000 7ff096ffd700 1 --2- 172.21.15.91:0/3572858183 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff08006b130 0x7ff08006d5f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff08c009c10 tx=0x7ff08c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.605+0000 7f09fe59c700 1 --2- 172.21.15.91:0/1623465741 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f09e006aed0 0x7f09e006d390 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f09f4006fd0 tx=0x7f09f4006bc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.606+0000 7f81d67fc700 1 -- 172.21.15.91:0/586856665 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c800b860 con 0x7f81d8071390 2023-11-24T06:08:09.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.606+0000 7f81d67fc700 1 -- 172.21.15.91:0/586856665 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81c800b9c0 con 0x7f81d8071390 2023-11-24T06:08:09.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.606+0000 7f81d67fc700 1 -- 172.21.15.91:0/586856665 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81c800bd30 con 0x7f81d8071390 2023-11-24T06:08:09.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.606+0000 7ff0337fe700 1 -- 172.21.15.91:0/511986322 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff024004230 con 0x7ff0340b08f0 2023-11-24T06:08:09.629 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.606+0000 7ff07f7fe700 1 -- 172.21.15.91:0/3572858183 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff088053b30 con 0x7ff0980b4d50 2023-11-24T06:08:09.629 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.606+0000 7ff0337fe700 1 -- 172.21.15.91:0/511986322 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff024004390 con 0x7ff0340b08f0 2023-11-24T06:08:09.629 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.606+0000 7ff0337fe700 1 -- 172.21.15.91:0/511986322 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0240036a0 con 0x7ff0340b08f0 2023-11-24T06:08:09.629 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.607+0000 7fd565ffb700 1 -- 172.21.15.91:0/3358550477 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5701682f0 con 0x7fd57006eef0 2023-11-24T06:08:09.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.607+0000 7f81ddff9700 1 -- 172.21.15.91:0/586856665 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f81d8071390 msgr2=0x7f81d815a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.607+0000 7f81ddff9700 1 --2- 172.21.15.91:0/586856665 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f81d8071390 0x7f81d815a520 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f81c80098f0 tx=0x7f81c8009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:09.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.607+0000 7f81ddff9700 1 -- 172.21.15.91:0/586856665 shutdown_connections 2023-11-24T06:08:09.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.607+0000 7f81ddff9700 1 --2- 172.21.15.91:0/586856665 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f81d8071390 0x7f81d815a520 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.607+0000 7f81ddff9700 1 --2- 172.21.15.91:0/586856665 >> v2:172.21.15.94:3300/0 conn(0x7f81d80d04a0 0x7f81d815aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.607+0000 7f81ddff9700 1 -- 172.21.15.91:0/586856665 >> 172.21.15.91:0/586856665 conn(0x7f81d80cf980 msgr2=0x7f81d80cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:09.631 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.607+0000 7ff03aead700 1 -- 172.21.15.91:0/511986322 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff0340b08f0 msgr2=0x7ff0340d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.631 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.607+0000 7ff03aead700 1 --2- 172.21.15.91:0/511986322 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff0340b08f0 0x7ff0340d0230 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff0340afe50 tx=0x7ff024009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.631 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.607+0000 7f81ddff9700 1 -- 172.21.15.91:0/586856665 shutdown_connections 2023-11-24T06:08:09.631 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.608+0000 7f81ddff9700 1 -- 172.21.15.91:0/586856665 wait complete. 2023-11-24T06:08:09.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.608+0000 7ff03aead700 1 -- 172.21.15.91:0/511986322 shutdown_connections 2023-11-24T06:08:09.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.608+0000 7ff03aead700 1 --2- 172.21.15.91:0/511986322 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff0340b08f0 0x7ff0340d0230 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.608+0000 7ff03aead700 1 --2- 172.21.15.91:0/511986322 >> v2:172.21.15.94:3300/0 conn(0x7ff03406c5d0 0x7ff0340b03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.608+0000 7ff03aead700 1 -- 172.21.15.91:0/511986322 >> 172.21.15.91:0/511986322 conn(0x7ff0340717f0 msgr2=0x7ff0340b5070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:09.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.608+0000 7ff03aead700 1 -- 172.21.15.91:0/511986322 shutdown_connections 2023-11-24T06:08:09.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.608+0000 7ff03aead700 1 -- 172.21.15.91:0/511986322 wait complete. 2023-11-24T06:08:09.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.609+0000 7f09eeffd700 1 -- 172.21.15.91:0/1623465741 <== mon.1 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a000b4e50 con 0x7f0a00145870 2023-11-24T06:08:09.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.609+0000 7ff03aead700 1 Processor -- start 2023-11-24T06:08:09.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.609+0000 7f81ddff9700 1 Processor -- start 2023-11-24T06:08:09.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.610+0000 7ff03aead700 1 -- start start 2023-11-24T06:08:09.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.611+0000 7ff03aead700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff03406c5d0 0x7ff03414b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.611+0000 7ff03aead700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0340b08f0 0x7ff03414aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.611+0000 7ff03aead700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff03415afc0 0x7ff03409d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.611+0000 7ff03aead700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03406e800 con 0x7ff03406c5d0 2023-11-24T06:08:09.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.611+0000 7ff03aead700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0341f9270 con 0x7ff03415afc0 2023-11-24T06:08:09.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.611+0000 7ff03aead700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff03406d9a0 con 0x7ff0340b08f0 2023-11-24T06:08:09.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.611+0000 7f81ddff9700 1 -- start start 2023-11-24T06:08:09.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.612+0000 7f81ddff9700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f81d8071390 0x7f81d806eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.612+0000 7f81ddff9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f81d80d04a0 0x7f81d8133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.612+0000 7f81ddff9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81d815afa0 0x7f81d81340f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.612+0000 7f81ddff9700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d80b3630 con 0x7f81d8071390 2023-11-24T06:08:09.635 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.612+0000 7f81ddff9700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81d80b2740 con 0x7f81d815afa0 2023-11-24T06:08:09.635 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.612+0000 7f81ddff9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81d80b2a20 con 0x7f81d80d04a0 2023-11-24T06:08:09.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.614+0000 7ff03944a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff03415afc0 0x7ff03409d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.614+0000 7ff033fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0340b08f0 0x7ff03414aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.614+0000 7ff03944a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff03415afc0 0x7ff03409d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:42554/0 (socket says 172.21.15.91:42554) 2023-11-24T06:08:09.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.614+0000 7ff03944a700 1 -- 172.21.15.91:0/463395873 learned_addr learned my addr 172.21.15.91:0/463395873 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:09.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.614+0000 7ff03944a700 1 -- 172.21.15.91:0/463395873 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0340b08f0 msgr2=0x7ff03414aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.614+0000 7ff03944a700 1 --2- 172.21.15.91:0/463395873 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0340b08f0 0x7ff03414aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.614+0000 7ff03944a700 1 -- 172.21.15.91:0/463395873 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff03406c5d0 msgr2=0x7ff03414b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.614+0000 7ff03944a700 1 --2- 172.21.15.91:0/463395873 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff03406c5d0 0x7ff03414b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.614+0000 7ff03944a700 1 -- 172.21.15.91:0/463395873 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff024009550 con 0x7ff03415afc0 2023-11-24T06:08:09.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.614+0000 7ff03944a700 1 --2- 172.21.15.91:0/463395873 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff03415afc0 0x7ff03409d550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff034070ab0 tx=0x7ff024004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.618+0000 7f81d6ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f81d80d04a0 0x7f81d8133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.618+0000 7f81d6ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f81d80d04a0 0x7f81d8133bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:41598/0 (socket says 172.21.15.91:41598) 2023-11-24T06:08:09.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.618+0000 7f81d6ffd700 1 -- 172.21.15.91:0/4141244051 learned_addr learned my addr 172.21.15.91:0/4141244051 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:09.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.618+0000 7f81d6ffd700 1 -- 172.21.15.91:0/4141244051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81d815afa0 msgr2=0x7f81d81340f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-24T06:08:09.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.618+0000 7f81d6ffd700 1 --2- 172.21.15.91:0/4141244051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81d815afa0 0x7f81d81340f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.618+0000 7f81d6ffd700 1 -- 172.21.15.91:0/4141244051 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f81d8071390 msgr2=0x7f81d806eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.618+0000 7f81d6ffd700 1 --2- 172.21.15.91:0/4141244051 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f81d8071390 0x7f81d806eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.618+0000 7f81d6ffd700 1 -- 172.21.15.91:0/4141244051 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c8009550 con 0x7f81d80d04a0 2023-11-24T06:08:09.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.618+0000 7f81d6ffd700 1 --2- 172.21.15.91:0/4141244051 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f81d80d04a0 0x7f81d8133bb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00e810 tx=0x7f81cc00ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.621+0000 7ff0317fa700 1 -- 172.21.15.91:0/463395873 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff024011630 con 0x7ff03415afc0 2023-11-24T06:08:09.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.621+0000 7ff0317fa700 1 -- 172.21.15.91:0/463395873 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0240117b0 con 0x7ff03415afc0 2023-11-24T06:08:09.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.621+0000 7ff0317fa700 1 -- 172.21.15.91:0/463395873 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff024011b60 con 0x7ff03415afc0 2023-11-24T06:08:09.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.621+0000 7f81bffff700 1 -- 172.21.15.91:0/4141244051 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc013610 con 0x7f81d80d04a0 2023-11-24T06:08:09.640 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.621+0000 7f81bffff700 1 -- 172.21.15.91:0/4141244051 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81cc013770 con 0x7f81d80d04a0 2023-11-24T06:08:09.640 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.621+0000 7f81bffff700 1 -- 172.21.15.91:0/4141244051 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81cc013b20 con 0x7f81d80d04a0 2023-11-24T06:08:09.640 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.621+0000 7ff03aead700 1 -- 172.21.15.91:0/463395873 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff03406dc80 con 0x7ff03415afc0 2023-11-24T06:08:09.640 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.621+0000 7ff03aead700 1 -- 172.21.15.91:0/463395873 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff03406e240 con 0x7ff03415afc0 2023-11-24T06:08:09.640 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.622+0000 7f81ddff9700 1 -- 172.21.15.91:0/4141244051 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81d80b2d00 con 0x7f81d80d04a0 2023-11-24T06:08:09.640 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.622+0000 7f81ddff9700 1 -- 172.21.15.91:0/4141244051 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81d80b32c0 con 0x7f81d80d04a0 2023-11-24T06:08:09.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.623+0000 7ff03aead700 1 -- 172.21.15.91:0/463395873 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0140052e0 con 0x7ff03415afc0 2023-11-24T06:08:09.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.623+0000 7f81ddff9700 1 -- 172.21.15.91:0/4141244051 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81d80b3480 con 0x7f81d80d04a0 2023-11-24T06:08:09.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.624+0000 7f81bffff700 1 -- 172.21.15.91:0/4141244051 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f81cc013c80 con 0x7f81d80d04a0 2023-11-24T06:08:09.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.625+0000 7f81bffff700 1 --2- 172.21.15.91:0/4141244051 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f81c406afa0 0x7f81c406d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.625+0000 7f81bffff700 1 -- 172.21.15.91:0/4141244051 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81cc0892a0 con 0x7f81d80d04a0 2023-11-24T06:08:09.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.625+0000 7ff0317fa700 1 -- 172.21.15.91:0/463395873 <== mon.1 v2:172.21.15.94:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff024022070 con 0x7ff03415afc0 2023-11-24T06:08:09.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.626+0000 7ff0317fa700 1 --2- 172.21.15.91:0/463395873 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff02006afa0 0x7ff02006d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.626+0000 7ff0317fa700 1 -- 172.21.15.91:0/463395873 <== mon.1 v2:172.21.15.94:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff02408b280 con 0x7ff03415afc0 2023-11-24T06:08:09.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.626+0000 7ff038c49700 1 --2- 172.21.15.91:0/463395873 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff02006afa0 0x7ff02006d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.626+0000 7f81d77fe700 1 --2- 172.21.15.91:0/4141244051 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f81c406afa0 0x7f81c406d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.627+0000 7ff038c49700 1 --2- 172.21.15.91:0/463395873 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff02006afa0 0x7ff02006d460 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff0280096d0 tx=0x7ff028006c30 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.627+0000 7f81d77fe700 1 --2- 172.21.15.91:0/4141244051 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f81c406afa0 0x7f81c406d460 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f81c80098f0 tx=0x7f81c8010000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.628+0000 7ff0317fa700 1 -- 172.21.15.91:0/463395873 <== mon.1 v2:172.21.15.94:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff0240564c0 con 0x7ff03415afc0 2023-11-24T06:08:09.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.629+0000 7f81bffff700 1 -- 172.21.15.91:0/4141244051 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f81cc054340 con 0x7f81d80d04a0 2023-11-24T06:08:09.670 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.663+0000 7fdb703eb700 1 Processor -- start 2023-11-24T06:08:09.670 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.664+0000 7fdb703eb700 1 -- start start 2023-11-24T06:08:09.670 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.665+0000 7fdb703eb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdb68070a80 0x7fdb68070e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.670 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.665+0000 7fdb703eb700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb680ca8e0 con 0x7fdb68145870 2023-11-24T06:08:09.670 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.665+0000 7fdb703eb700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb680713a0 con 0x7fdb68166600 2023-11-24T06:08:09.670 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.665+0000 7fdb703eb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb68166a50 con 0x7fdb68070a80 2023-11-24T06:08:09.670 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.666+0000 7fdb6e988700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fdb68166600 0x7fdb680cad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.91:54898/0 (socket says 172.21.15.91:54898) 2023-11-24T06:08:09.671 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.666+0000 7fdb6e988700 1 -- 172.21.15.91:0/3919109415 learned_addr learned my addr 172.21.15.91:0/3919109415 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:09.671 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.666+0000 7fdb6e187700 1 --2- 172.21.15.91:0/3919109415 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdb68070a80 0x7fdb68070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.672 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.667+0000 7fdb6d185700 1 -- 172.21.15.91:0/3919109415 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580138974 0 0) 0x7fdb680713a0 con 0x7fdb68166600 2023-11-24T06:08:09.672 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.667+0000 7fdb6d185700 1 -- 172.21.15.91:0/3919109415 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb540035b0 con 0x7fdb68166600 2023-11-24T06:08:09.672 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.667+0000 7fdb6d185700 1 -- 172.21.15.91:0/3919109415 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910482844 0 0) 0x7fdb680ca8e0 con 0x7fdb68145870 2023-11-24T06:08:09.673 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.667+0000 7fdb6d185700 1 -- 172.21.15.91:0/3919109415 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb680713a0 con 0x7fdb68145870 2023-11-24T06:08:09.673 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.668+0000 7fdb6e187700 1 -- 172.21.15.91:0/3919109415 >> v1:172.21.15.94:6789/0 conn(0x7fdb68166600 legacy=0x7fdb680cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.673 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.668+0000 7fdb6d185700 1 -- 172.21.15.91:0/3919109415 <== mon.1 v1:172.21.15.94:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2012221691 0 0) 0x7fdb540035b0 con 0x7fdb68166600 2023-11-24T06:08:09.673 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.668+0000 7fdb6e187700 1 -- 172.21.15.91:0/3919109415 >> v1:172.21.15.91:6789/0 conn(0x7fdb68145870 legacy=0x7fdb68165ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.673 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.668+0000 7fdb6e187700 1 -- 172.21.15.91:0/3919109415 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb680c9fd0 con 0x7fdb68070a80 2023-11-24T06:08:09.673 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.669+0000 7fdb6e187700 1 --2- 172.21.15.91:0/3919109415 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdb68070a80 0x7fdb68070e60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdb580097f0 tx=0x7fdb58009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4754a86af10388 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.674 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.669+0000 7fdb6d185700 1 -- 172.21.15.91:0/3919109415 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb60001a30 con 0x7fdb68070a80 2023-11-24T06:08:09.674 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.669+0000 7fdb6d185700 1 -- 172.21.15.91:0/3919109415 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb58004030 con 0x7fdb68070a80 2023-11-24T06:08:09.674 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.669+0000 7fdb6d185700 1 -- 172.21.15.91:0/3919109415 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb580043c0 con 0x7fdb68070a80 2023-11-24T06:08:09.674 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.670+0000 7fdb703eb700 1 -- 172.21.15.91:0/3919109415 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdb68070a80 msgr2=0x7fdb68070e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.674 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.670+0000 7fdb703eb700 1 --2- 172.21.15.91:0/3919109415 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdb68070a80 0x7fdb68070e60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdb580097f0 tx=0x7fdb58009b00 comp rx=0 tx=0).stop 2023-11-24T06:08:09.675 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.670+0000 7fdb703eb700 1 -- 172.21.15.91:0/3919109415 shutdown_connections 2023-11-24T06:08:09.675 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.670+0000 7fdb703eb700 1 --2- 172.21.15.91:0/3919109415 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdb68070a80 0x7fdb68070e60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.675 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.670+0000 7fdb703eb700 1 -- 172.21.15.91:0/3919109415 >> 172.21.15.91:0/3919109415 conn(0x7fdb680cf980 msgr2=0x7fdb680cfd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:09.675 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.671+0000 7fdb703eb700 1 -- 172.21.15.91:0/3919109415 shutdown_connections 2023-11-24T06:08:09.676 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.671+0000 7fdb703eb700 1 -- 172.21.15.91:0/3919109415 wait complete. 2023-11-24T06:08:09.677 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.673+0000 7fdb703eb700 1 Processor -- start 2023-11-24T06:08:09.678 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.673+0000 7fdb703eb700 1 -- start start 2023-11-24T06:08:09.678 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.674+0000 7fdb703eb700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb68070a80 0x7fdb6813f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.679 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.674+0000 7fdb703eb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdb68145870 0x7fdb68143410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.679 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.674+0000 7fdb703eb700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb68166600 0x7fdb680cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.679 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.674+0000 7fdb703eb700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb68143df0 con 0x7fdb68070a80 2023-11-24T06:08:09.679 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.674+0000 7fdb703eb700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb681440d0 con 0x7fdb68166600 2023-11-24T06:08:09.679 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.674+0000 7fdb703eb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb681443b0 con 0x7fdb68145870 2023-11-24T06:08:09.679 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.675+0000 7fdb6e187700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb68070a80 0x7fdb6813f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.679 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.675+0000 7fdb6e187700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb68070a80 0x7fdb6813f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58180/0 (socket says 172.21.15.91:58180) 2023-11-24T06:08:09.679 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.675+0000 7fdb6e187700 1 -- 172.21.15.91:0/2087026016 learned_addr learned my addr 172.21.15.91:0/2087026016 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:09.680 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.675+0000 7fdb6e187700 1 -- 172.21.15.91:0/2087026016 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdb68145870 msgr2=0x7fdb68143410 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:09.680 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.675+0000 7fdb6e187700 1 --2- 172.21.15.91:0/2087026016 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdb68145870 0x7fdb68143410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.680 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.675+0000 7fdb6e187700 1 -- 172.21.15.91:0/2087026016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb68166600 msgr2=0x7fdb680cd530 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-11-24T06:08:09.680 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.675+0000 7fdb6e187700 1 --2- 172.21.15.91:0/2087026016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb68166600 0x7fdb680cd530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.680 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.675+0000 7fdb6e187700 1 -- 172.21.15.91:0/2087026016 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb580094a0 con 0x7fdb68070a80 2023-11-24T06:08:09.680 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.676+0000 7fdb6e187700 1 --2- 172.21.15.91:0/2087026016 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb68070a80 0x7fdb6813f630 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdb58000c00 tx=0x7fdb58004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.682 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.677+0000 7fdb5effd700 1 -- 172.21.15.91:0/2087026016 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb58003b80 con 0x7fdb68070a80 2023-11-24T06:08:09.682 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.677+0000 7fdb5effd700 1 -- 172.21.15.91:0/2087026016 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5801f070 con 0x7fdb68070a80 2023-11-24T06:08:09.682 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.677+0000 7fdb5effd700 1 -- 172.21.15.91:0/2087026016 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb58011630 con 0x7fdb68070a80 2023-11-24T06:08:09.684 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.677+0000 7fdb703eb700 1 -- 172.21.15.91:0/2087026016 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb68144690 con 0x7fdb68070a80 2023-11-24T06:08:09.684 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.677+0000 7fdb703eb700 1 -- 172.21.15.91:0/2087026016 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6813fb70 con 0x7fdb68070a80 2023-11-24T06:08:09.686 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.682+0000 7fdb5effd700 1 -- 172.21.15.91:0/2087026016 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdb58021030 con 0x7fdb68070a80 2023-11-24T06:08:09.687 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.682+0000 7fdb5effd700 1 --2- 172.21.15.91:0/2087026016 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdb5406b470 0x7fdb5406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:09.687 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.683+0000 7fdb5effd700 1 -- 172.21.15.91:0/2087026016 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb5808a860 con 0x7fdb68070a80 2023-11-24T06:08:09.687 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.683+0000 7fdb6d986700 1 --2- 172.21.15.91:0/2087026016 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdb5406b470 0x7fdb5406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:09.688 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.684+0000 7fdb6d986700 1 --2- 172.21.15.91:0/2087026016 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdb5406b470 0x7fdb5406d930 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdb6815ecd0 tx=0x7fdb64007a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:09.689 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.685+0000 7fdb703eb700 1 -- 172.21.15.91:0/2087026016 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb500052e0 con 0x7fdb68070a80 2023-11-24T06:08:09.698 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.695+0000 7fdb5effd700 1 -- 172.21.15.91:0/2087026016 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdb58055980 con 0x7fdb68070a80 2023-11-24T06:08:09.983 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.979+0000 7feb7c37b700 1 -- 172.21.15.91:0/3914335925 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7feb740ca0a0 con 0x7feb7415db10 2023-11-24T06:08:09.983 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.980+0000 7feb6affd700 1 -- 172.21.15.91:0/3914335925 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feb6c057c70 con 0x7feb7415db10 2023-11-24T06:08:09.987 INFO:teuthology.orchestra.run.smithi091.stdout:55834574854 2023-11-24T06:08:09.987 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.983+0000 7feb7c37b700 1 -- 172.21.15.91:0/3914335925 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7feb5806afa0 msgr2=0x7feb5806d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.987 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.983+0000 7feb7c37b700 1 --2- 172.21.15.91:0/3914335925 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7feb5806afa0 0x7feb5806d460 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7feb6400d370 tx=0x7feb6400b040 comp rx=0 tx=0).stop 2023-11-24T06:08:09.987 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.983+0000 7feb7c37b700 1 -- 172.21.15.91:0/3914335925 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feb7415db10 msgr2=0x7feb74138d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:09.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.983+0000 7feb7c37b700 1 --2- 172.21.15.91:0/3914335925 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feb7415db10 0x7feb74138d30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00b990 tx=0x7feb6c00bca0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.984+0000 7feb7c37b700 1 -- 172.21.15.91:0/3914335925 shutdown_connections 2023-11-24T06:08:09.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.984+0000 7feb7c37b700 1 --2- 172.21.15.91:0/3914335925 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7feb5806afa0 0x7feb5806d460 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7feb6400d370 tx=0x7feb6400b040 comp rx=0 tx=0).stop 2023-11-24T06:08:09.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.984+0000 7feb7c37b700 1 --2- 172.21.15.91:0/3914335925 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feb7415db10 0x7feb74138d30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.984+0000 7feb7c37b700 1 --2- 172.21.15.91:0/3914335925 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feb74071390 0x7feb740b0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.984+0000 7feb7c37b700 1 --2- 172.21.15.91:0/3914335925 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feb740ca2c0 0x7feb741387f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:09.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.984+0000 7feb7c37b700 1 -- 172.21.15.91:0/3914335925 >> 172.21.15.91:0/3914335925 conn(0x7feb740cf980 msgr2=0x7feb7406ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:09.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.985+0000 7feb7c37b700 1 -- 172.21.15.91:0/3914335925 shutdown_connections 2023-11-24T06:08:09.989 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:09.985+0000 7feb7c37b700 1 -- 172.21.15.91:0/3914335925 wait complete. 2023-11-24T06:08:10.007 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2023-11-24T06:08:10.007 DEBUG:teuthology.parallel:result is None 2023-11-24T06:08:10.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.004+0000 7ff09d2ce700 1 -- 172.21.15.91:0/3572858183 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7ff078005590 con 0x7ff0980b4d50 2023-11-24T06:08:10.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.005+0000 7f9a62d9c700 1 -- 172.21.15.91:0/797809376 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9a5c067f60 con 0x7f9a5c070a80 2023-11-24T06:08:10.015 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.012+0000 7ff07f7fe700 1 -- 172.21.15.91:0/3572858183 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff088017020 con 0x7ff0980b4d50 2023-11-24T06:08:10.016 INFO:teuthology.orchestra.run.smithi091.stdout:55834574856 2023-11-24T06:08:10.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.012+0000 7f9a597fa700 1 -- 172.21.15.91:0/797809376 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9a4c054510 con 0x7f9a5c070a80 2023-11-24T06:08:10.018 INFO:teuthology.orchestra.run.smithi091.stdout:55834574855 2023-11-24T06:08:10.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.014+0000 7ff09d2ce700 1 -- 172.21.15.91:0/3572858183 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff08006b130 msgr2=0x7ff08006d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.014+0000 7ff09d2ce700 1 --2- 172.21.15.91:0/3572858183 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff08006b130 0x7ff08006d5f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff08c009c10 tx=0x7ff08c008030 comp rx=0 tx=0).stop 2023-11-24T06:08:10.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.014+0000 7ff09d2ce700 1 -- 172.21.15.91:0/3572858183 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff0980b4d50 msgr2=0x7ff09815c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.014+0000 7ff09d2ce700 1 --2- 172.21.15.91:0/3572858183 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff0980b4d50 0x7ff09815c2a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff08800c4e0 tx=0x7ff08800c8a0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.014+0000 7ff09d2ce700 1 -- 172.21.15.91:0/3572858183 shutdown_connections 2023-11-24T06:08:10.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.014+0000 7ff09d2ce700 1 --2- 172.21.15.91:0/3572858183 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff08006b130 0x7ff08006d5f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.014+0000 7ff09d2ce700 1 --2- 172.21.15.91:0/3572858183 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff09815abb0 0x7ff0980ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.014+0000 7ff09d2ce700 1 --2- 172.21.15.91:0/3572858183 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0980724a0 0x7ff098149e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.014+0000 7ff09d2ce700 1 --2- 172.21.15.91:0/3572858183 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff0980b4d50 0x7ff09815c2a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.014+0000 7ff09d2ce700 1 -- 172.21.15.91:0/3572858183 >> 172.21.15.91:0/3572858183 conn(0x7ff0980d0aa0 msgr2=0x7ff0981556a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:10.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.015+0000 7ff09d2ce700 1 -- 172.21.15.91:0/3572858183 shutdown_connections 2023-11-24T06:08:10.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.015+0000 7fd577791700 1 -- 172.21.15.91:0/3358550477 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fd5701682f0 con 0x7fd57006eef0 2023-11-24T06:08:10.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.015+0000 7ff09d2ce700 1 -- 172.21.15.91:0/3572858183 wait complete. 2023-11-24T06:08:10.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.015+0000 7fd565ffb700 1 -- 172.21.15.91:0/3358550477 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd5701682f0 con 0x7fd57006eef0 2023-11-24T06:08:10.020 INFO:teuthology.orchestra.run.smithi091.stdout:55834574855 2023-11-24T06:08:10.021 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.016+0000 7f9a62d9c700 1 -- 172.21.15.91:0/797809376 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9a4806aed0 msgr2=0x7f9a4806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.021 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.016+0000 7f9a62d9c700 1 --2- 172.21.15.91:0/797809376 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9a4806aed0 0x7f9a4806d390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c144850 tx=0x7f9a50007ec0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.021 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.016+0000 7f9a62d9c700 1 -- 172.21.15.91:0/797809376 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9a5c070a80 msgr2=0x7f9a5c143d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.021 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.016+0000 7f9a62d9c700 1 --2- 172.21.15.91:0/797809376 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9a5c070a80 0x7f9a5c143d70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c00b450 tx=0x7f9a4c00b760 comp rx=0 tx=0).stop 2023-11-24T06:08:10.021 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.017+0000 7f9a62d9c700 1 -- 172.21.15.91:0/797809376 shutdown_connections 2023-11-24T06:08:10.021 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.017+0000 7f9a62d9c700 1 --2- 172.21.15.91:0/797809376 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9a4806aed0 0x7f9a4806d390 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.021 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.017+0000 7f9a62d9c700 1 --2- 172.21.15.91:0/797809376 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9a5c166600 0x7f9a5c13fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.021 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.017+0000 7f9a62d9c700 1 --2- 172.21.15.91:0/797809376 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9a5c070a80 0x7f9a5c143d70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.017+0000 7f9a62d9c700 1 --2- 172.21.15.91:0/797809376 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9a5c145870 0x7f9a5c0b48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.017+0000 7f9a62d9c700 1 -- 172.21.15.91:0/797809376 >> 172.21.15.91:0/797809376 conn(0x7f9a5c0cf980 msgr2=0x7f9a5c06eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:10.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.018+0000 7fd577791700 1 -- 172.21.15.91:0/3358550477 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd55406af20 msgr2=0x7fd55406d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.018+0000 7fd577791700 1 --2- 172.21.15.91:0/3358550477 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd55406af20 0x7fd55406d3e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd57013ad70 tx=0x7fd56c008030 comp rx=0 tx=0).stop 2023-11-24T06:08:10.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.018+0000 7fd577791700 1 -- 172.21.15.91:0/3358550477 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd57006eef0 msgr2=0x7fd5700cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.018+0000 7fd577791700 1 --2- 172.21.15.91:0/3358550477 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd57006eef0 0x7fd5700cf140 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd56000b470 tx=0x7fd56000b830 comp rx=0 tx=0).stop 2023-11-24T06:08:10.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.021+0000 7fd577791700 1 -- 172.21.15.91:0/3358550477 shutdown_connections 2023-11-24T06:08:10.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.021+0000 7fd577791700 1 --2- 172.21.15.91:0/3358550477 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd55406af20 0x7fd55406d3e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.021+0000 7fd577791700 1 --2- 172.21.15.91:0/3358550477 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd570145700 0x7fd570139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.021+0000 7fd577791700 1 --2- 172.21.15.91:0/3358550477 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd57006eef0 0x7fd5700cf140 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.021+0000 7fd577791700 1 --2- 172.21.15.91:0/3358550477 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd570071390 0x7fd57016a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.021+0000 7fd577791700 1 -- 172.21.15.91:0/3358550477 >> 172.21.15.91:0/3358550477 conn(0x7fd5700cf980 msgr2=0x7fd5700b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:10.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.021+0000 7f0a049ef700 1 -- 172.21.15.91:0/1623465741 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f0a000b4e50 con 0x7f0a00145870 2023-11-24T06:08:10.026 INFO:teuthology.orchestra.run.smithi091.stdout:55834574855 2023-11-24T06:08:10.026 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.022+0000 7f09eeffd700 1 -- 172.21.15.91:0/1623465741 <== mon.1 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0a000b4e50 con 0x7f0a00145870 2023-11-24T06:08:10.026 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.022+0000 7fd577791700 1 -- 172.21.15.91:0/3358550477 shutdown_connections 2023-11-24T06:08:10.026 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.022+0000 7fd577791700 1 -- 172.21.15.91:0/3358550477 wait complete. 2023-11-24T06:08:10.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.024+0000 7f9a62d9c700 1 -- 172.21.15.91:0/797809376 shutdown_connections 2023-11-24T06:08:10.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.024+0000 7f9a62d9c700 1 -- 172.21.15.91:0/797809376 wait complete. 2023-11-24T06:08:10.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.024+0000 7ff03aead700 1 -- 172.21.15.91:0/463395873 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ff014005860 con 0x7ff03415afc0 2023-11-24T06:08:10.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.024+0000 7f0a049ef700 1 -- 172.21.15.91:0/1623465741 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f09e006aed0 msgr2=0x7f09e006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.024+0000 7f0a049ef700 1 --2- 172.21.15.91:0/1623465741 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f09e006aed0 0x7f09e006d390 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f09f4006fd0 tx=0x7f09f4006bc0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.024+0000 7f0a049ef700 1 -- 172.21.15.91:0/1623465741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a00145870 msgr2=0x7f0a000b5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.024+0000 7f0a049ef700 1 --2- 172.21.15.91:0/1623465741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a00145870 0x7f0a000b5130 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f09f000b470 tx=0x7f09f000b780 comp rx=0 tx=0).stop 2023-11-24T06:08:10.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.025+0000 7ff0317fa700 1 -- 172.21.15.91:0/463395873 <== mon.1 v2:172.21.15.94:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff024059ae0 con 0x7ff03415afc0 2023-11-24T06:08:10.029 INFO:teuthology.orchestra.run.smithi091.stdout:55834574856 2023-11-24T06:08:10.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.027+0000 7f81ddff9700 1 -- 172.21.15.91:0/4141244051 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f81d80b3770 con 0x7f81d80d04a0 2023-11-24T06:08:10.031 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.027+0000 7f81bffff700 1 -- 172.21.15.91:0/4141244051 <== mon.2 v2:172.21.15.94:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f81d80b3770 con 0x7f81d80d04a0 2023-11-24T06:08:10.031 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.026+0000 7f0a049ef700 1 -- 172.21.15.91:0/1623465741 shutdown_connections 2023-11-24T06:08:10.031 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.026+0000 7f0a049ef700 1 --2- 172.21.15.91:0/1623465741 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f09e006aed0 0x7f09e006d390 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.031 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.026+0000 7f0a049ef700 1 --2- 172.21.15.91:0/1623465741 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0a00166600 0x7f0a000afd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.032 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.026+0000 7f0a049ef700 1 --2- 172.21.15.91:0/1623465741 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0a00070a80 0x7f0a0013a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.032 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.026+0000 7f0a049ef700 1 --2- 172.21.15.91:0/1623465741 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0a00145870 0x7f0a000b5130 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f09f000b470 tx=0x7f09f000b780 comp rx=0 tx=0).stop 2023-11-24T06:08:10.032 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.026+0000 7f0a049ef700 1 -- 172.21.15.91:0/1623465741 >> 172.21.15.91:0/1623465741 conn(0x7f0a000cf980 msgr2=0x7f0a000cd530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:10.032 INFO:teuthology.orchestra.run.smithi091.stdout:55834574856 2023-11-24T06:08:10.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.029+0000 7ff03aead700 1 -- 172.21.15.91:0/463395873 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff02006afa0 msgr2=0x7ff02006d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.029+0000 7ff03aead700 1 --2- 172.21.15.91:0/463395873 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff02006afa0 0x7ff02006d460 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff0280096d0 tx=0x7ff028006c30 comp rx=0 tx=0).stop 2023-11-24T06:08:10.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.029+0000 7ff03aead700 1 -- 172.21.15.91:0/463395873 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff03415afc0 msgr2=0x7ff03409d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.029+0000 7ff03aead700 1 --2- 172.21.15.91:0/463395873 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff03415afc0 0x7ff03409d550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff034070ab0 tx=0x7ff024004750 comp rx=0 tx=0).stop 2023-11-24T06:08:10.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.029+0000 7ff03aead700 1 -- 172.21.15.91:0/463395873 shutdown_connections 2023-11-24T06:08:10.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.029+0000 7ff03aead700 1 --2- 172.21.15.91:0/463395873 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff02006afa0 0x7ff02006d460 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.029+0000 7ff03aead700 1 --2- 172.21.15.91:0/463395873 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff03415afc0 0x7ff03409d550 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.029+0000 7ff03aead700 1 --2- 172.21.15.91:0/463395873 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff03406c5d0 0x7ff03414b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.029+0000 7ff03aead700 1 --2- 172.21.15.91:0/463395873 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff0340b08f0 0x7ff03414aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.029+0000 7ff03aead700 1 -- 172.21.15.91:0/463395873 >> 172.21.15.91:0/463395873 conn(0x7ff0340717f0 msgr2=0x7ff0340b1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:10.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.029+0000 7ff03aead700 1 -- 172.21.15.91:0/463395873 shutdown_connections 2023-11-24T06:08:10.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.029+0000 7ff03aead700 1 -- 172.21.15.91:0/463395873 wait complete. 2023-11-24T06:08:10.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.030+0000 7f0a049ef700 1 -- 172.21.15.91:0/1623465741 shutdown_connections 2023-11-24T06:08:10.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.030+0000 7f0a049ef700 1 -- 172.21.15.91:0/1623465741 wait complete. 2023-11-24T06:08:10.035 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.031+0000 7f81ddff9700 1 -- 172.21.15.91:0/4141244051 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f81c406afa0 msgr2=0x7f81c406d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.031+0000 7f81ddff9700 1 --2- 172.21.15.91:0/4141244051 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f81c406afa0 0x7f81c406d460 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f81c80098f0 tx=0x7f81c8010000 comp rx=0 tx=0).stop 2023-11-24T06:08:10.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.031+0000 7f81ddff9700 1 -- 172.21.15.91:0/4141244051 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f81d80d04a0 msgr2=0x7f81d8133bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.031+0000 7f81ddff9700 1 --2- 172.21.15.91:0/4141244051 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f81d80d04a0 0x7f81d8133bb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00e810 tx=0x7f81cc00ebd0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.032+0000 7f81ddff9700 1 -- 172.21.15.91:0/4141244051 shutdown_connections 2023-11-24T06:08:10.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.032+0000 7f81ddff9700 1 --2- 172.21.15.91:0/4141244051 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f81c406afa0 0x7f81c406d460 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.032+0000 7f81ddff9700 1 --2- 172.21.15.91:0/4141244051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f81d815afa0 0x7f81d81340f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.032+0000 7f81ddff9700 1 --2- 172.21.15.91:0/4141244051 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f81d8071390 0x7f81d806eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.032+0000 7f81ddff9700 1 --2- 172.21.15.91:0/4141244051 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f81d80d04a0 0x7f81d8133bb0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.032+0000 7f81ddff9700 1 -- 172.21.15.91:0/4141244051 >> 172.21.15.91:0/4141244051 conn(0x7f81d80cf980 msgr2=0x7f81d806cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:10.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.032+0000 7f81ddff9700 1 -- 172.21.15.91:0/4141244051 shutdown_connections 2023-11-24T06:08:10.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.032+0000 7f81ddff9700 1 -- 172.21.15.91:0/4141244051 wait complete. 2023-11-24T06:08:10.039 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2023-11-24T06:08:10.039 DEBUG:teuthology.parallel:result is None 2023-11-24T06:08:10.046 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2023-11-24T06:08:10.046 DEBUG:teuthology.parallel:result is None 2023-11-24T06:08:10.047 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2023-11-24T06:08:10.048 DEBUG:teuthology.parallel:result is None 2023-11-24T06:08:10.057 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2023-11-24T06:08:10.057 DEBUG:teuthology.parallel:result is None 2023-11-24T06:08:10.058 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2023-11-24T06:08:10.058 DEBUG:teuthology.parallel:result is None 2023-11-24T06:08:10.059 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2023-11-24T06:08:10.059 DEBUG:teuthology.parallel:result is None 2023-11-24T06:08:10.079 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.076+0000 7fdb703eb700 1 -- 172.21.15.91:0/2087026016 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fdb50005590 con 0x7fdb68070a80 2023-11-24T06:08:10.080 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.076+0000 7fdb5effd700 1 -- 172.21.15.91:0/2087026016 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdb58058fa0 con 0x7fdb68070a80 2023-11-24T06:08:10.080 INFO:teuthology.orchestra.run.smithi091.stdout:55834574855 2023-11-24T06:08:10.083 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.079+0000 7fdb703eb700 1 -- 172.21.15.91:0/2087026016 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdb5406b470 msgr2=0x7fdb5406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.083 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.079+0000 7fdb703eb700 1 --2- 172.21.15.91:0/2087026016 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdb5406b470 0x7fdb5406d930 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdb6815ecd0 tx=0x7fdb64007a20 comp rx=0 tx=0).stop 2023-11-24T06:08:10.083 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.079+0000 7fdb703eb700 1 -- 172.21.15.91:0/2087026016 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb68070a80 msgr2=0x7fdb6813f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.083 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.079+0000 7fdb703eb700 1 --2- 172.21.15.91:0/2087026016 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb68070a80 0x7fdb6813f630 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdb58000c00 tx=0x7fdb58004750 comp rx=0 tx=0).stop 2023-11-24T06:08:10.083 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.079+0000 7fdb703eb700 1 -- 172.21.15.91:0/2087026016 shutdown_connections 2023-11-24T06:08:10.083 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.079+0000 7fdb703eb700 1 --2- 172.21.15.91:0/2087026016 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdb5406b470 0x7fdb5406d930 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.083 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.079+0000 7fdb703eb700 1 --2- 172.21.15.91:0/2087026016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb68166600 0x7fdb680cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.083 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.079+0000 7fdb703eb700 1 --2- 172.21.15.91:0/2087026016 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb68070a80 0x7fdb6813f630 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.079+0000 7fdb703eb700 1 --2- 172.21.15.91:0/2087026016 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdb68145870 0x7fdb68143410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.079+0000 7fdb703eb700 1 -- 172.21.15.91:0/2087026016 >> 172.21.15.91:0/2087026016 conn(0x7fdb680cf980 msgr2=0x7fdb680cead0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:10.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.079+0000 7fdb703eb700 1 -- 172.21.15.91:0/2087026016 shutdown_connections 2023-11-24T06:08:10.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.079+0000 7fdb703eb700 1 -- 172.21.15.91:0/2087026016 wait complete. 2023-11-24T06:08:10.094 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2023-11-24T06:08:10.094 DEBUG:teuthology.parallel:result is None 2023-11-24T06:08:10.095 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-24T06:08:10.095 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:08:10.254 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.250+0000 7f423cb30700 1 Processor -- start 2023-11-24T06:08:10.255 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.251+0000 7f423cb30700 1 -- start start 2023-11-24T06:08:10.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.251+0000 7f423cb30700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f423806b400 0x7f423815e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:10.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.251+0000 7f423cb30700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f42380b5e40 0x7f423815eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:10.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f423cb30700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423806bb80 con 0x7f423815e1e0 2023-11-24T06:08:10.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f423cb30700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f423806be20 con 0x7f42380b5e40 2023-11-24T06:08:10.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f423cb30700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423806c100 con 0x7f423806b400 2023-11-24T06:08:10.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f4236d9d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f42380b5e40 0x7f423815eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:10.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f423659c700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f423815e1e0 0x7f42380b12a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53120/0 (socket says 172.21.15.91:53120) 2023-11-24T06:08:10.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f4236d9d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f42380b5e40 0x7f423815eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58190/0 (socket says 172.21.15.91:58190) 2023-11-24T06:08:10.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f423659c700 1 -- 172.21.15.91:0/1960394097 learned_addr learned my addr 172.21.15.91:0/1960394097 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:10.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f4235d9b700 1 --2- 172.21.15.91:0/1960394097 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f423806b400 0x7f423815e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:10.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f4236d9d700 1 -- 172.21.15.91:0/1960394097 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f423806b400 msgr2=0x7f423815e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:10.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f4236d9d700 1 --2- 172.21.15.91:0/1960394097 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f423806b400 0x7f423815e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f4236d9d700 1 -- 172.21.15.91:0/1960394097 >> v1:172.21.15.91:6789/0 conn(0x7f423815e1e0 legacy=0x7f42380b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f4236d9d700 1 -- 172.21.15.91:0/1960394097 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423806c3e0 con 0x7f42380b5e40 2023-11-24T06:08:10.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f423559a700 1 -- 172.21.15.91:0/1960394097 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293909617 0 0) 0x7f423806bb80 con 0x7f423815e1e0 2023-11-24T06:08:10.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f4236d9d700 1 --2- 172.21.15.91:0/1960394097 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42380b5e40 0x7f423815eb00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4230009d90 tx=0x7f423000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d52be50994e77c65 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:10.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f423559a700 1 -- 172.21.15.91:0/1960394097 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423000e070 con 0x7f42380b5e40 2023-11-24T06:08:10.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f423559a700 1 -- 172.21.15.91:0/1960394097 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423000bbd0 con 0x7f42380b5e40 2023-11-24T06:08:10.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.252+0000 7f423559a700 1 -- 172.21.15.91:0/1960394097 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230012430 con 0x7f42380b5e40 2023-11-24T06:08:10.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.253+0000 7f423cb30700 1 -- 172.21.15.91:0/1960394097 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42380b5e40 msgr2=0x7f423815eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.259 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.253+0000 7f423cb30700 1 --2- 172.21.15.91:0/1960394097 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42380b5e40 0x7f423815eb00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4230009d90 tx=0x7f423000b330 comp rx=0 tx=0).stop 2023-11-24T06:08:10.259 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.253+0000 7f423cb30700 1 -- 172.21.15.91:0/1960394097 shutdown_connections 2023-11-24T06:08:10.259 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.253+0000 7f423cb30700 1 --2- 172.21.15.91:0/1960394097 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42380b5e40 0x7f423815eb00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.259 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.253+0000 7f423cb30700 1 --2- 172.21.15.91:0/1960394097 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f423806b400 0x7f423815e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.259 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.253+0000 7f423cb30700 1 -- 172.21.15.91:0/1960394097 >> 172.21.15.91:0/1960394097 conn(0x7f42380b5070 msgr2=0x7f42380b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:10.259 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.253+0000 7f423cb30700 1 -- 172.21.15.91:0/1960394097 shutdown_connections 2023-11-24T06:08:10.259 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.253+0000 7f423cb30700 1 -- 172.21.15.91:0/1960394097 wait complete. 2023-11-24T06:08:10.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.254+0000 7f423cb30700 1 Processor -- start 2023-11-24T06:08:10.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.255+0000 7f423cb30700 1 -- start start 2023-11-24T06:08:10.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.255+0000 7f423cb30700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f423806b400 0x7f42380cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:10.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.255+0000 7f423cb30700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42380b5e40 0x7f42381f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:10.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.255+0000 7f423cb30700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f423815e1e0 0x7f423809b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:10.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.255+0000 7f423cb30700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42380cd870 con 0x7f423806b400 2023-11-24T06:08:10.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.255+0000 7f423cb30700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42380cdb50 con 0x7f42380b5e40 2023-11-24T06:08:10.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.255+0000 7f423cb30700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42380cde30 con 0x7f423815e1e0 2023-11-24T06:08:10.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.256+0000 7f423659c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f423806b400 0x7f42380cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:10.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.256+0000 7f423659c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f423806b400 0x7f42380cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58192/0 (socket says 172.21.15.91:58192) 2023-11-24T06:08:10.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.256+0000 7f423659c700 1 -- 172.21.15.91:0/4254739949 learned_addr learned my addr 172.21.15.91:0/4254739949 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:10.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.256+0000 7f4236d9d700 1 --2- 172.21.15.91:0/4254739949 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f423815e1e0 0x7f423809b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:10.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.256+0000 7f4235d9b700 1 --2- 172.21.15.91:0/4254739949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42380b5e40 0x7f42381f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:10.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.256+0000 7f423659c700 1 -- 172.21.15.91:0/4254739949 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f423815e1e0 msgr2=0x7f423809b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.256+0000 7f423659c700 1 --2- 172.21.15.91:0/4254739949 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f423815e1e0 0x7f423809b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.256+0000 7f423659c700 1 -- 172.21.15.91:0/4254739949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42380b5e40 msgr2=0x7f42381f3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.256+0000 7f423659c700 1 --2- 172.21.15.91:0/4254739949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42380b5e40 0x7f42381f3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.256+0000 7f423659c700 1 -- 172.21.15.91:0/4254739949 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4230009a70 con 0x7f423806b400 2023-11-24T06:08:10.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.256+0000 7f423659c700 1 --2- 172.21.15.91:0/4254739949 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f423806b400 0x7f42380cd020 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f422c00c550 tx=0x7f422c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:10.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.256+0000 7f4226ffd700 1 -- 172.21.15.91:0/4254739949 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422c004c80 con 0x7f423806b400 2023-11-24T06:08:10.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.256+0000 7f4226ffd700 1 -- 172.21.15.91:0/4254739949 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422c004e00 con 0x7f423806b400 2023-11-24T06:08:10.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.256+0000 7f423cb30700 1 -- 172.21.15.91:0/4254739949 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42380ce110 con 0x7f423806b400 2023-11-24T06:08:10.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.256+0000 7f4226ffd700 1 -- 172.21.15.91:0/4254739949 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422c007310 con 0x7f423806b400 2023-11-24T06:08:10.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.256+0000 7f423cb30700 1 -- 172.21.15.91:0/4254739949 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42380cd560 con 0x7f423806b400 2023-11-24T06:08:10.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.257+0000 7f423cb30700 1 -- 172.21.15.91:0/4254739949 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4238068670 con 0x7f423806b400 2023-11-24T06:08:10.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.257+0000 7f4226ffd700 1 -- 172.21.15.91:0/4254739949 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f422c007470 con 0x7f423806b400 2023-11-24T06:08:10.264 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.258+0000 7f4226ffd700 1 --2- 172.21.15.91:0/4254739949 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f421806aed0 0x7f421806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:10.264 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.258+0000 7f4226ffd700 1 -- 172.21.15.91:0/4254739949 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f422c0892d0 con 0x7f423806b400 2023-11-24T06:08:10.264 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.258+0000 7f4235d9b700 1 --2- 172.21.15.91:0/4254739949 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f421806aed0 0x7f421806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:10.264 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.258+0000 7f4235d9b700 1 --2- 172.21.15.91:0/4254739949 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f421806aed0 0x7f421806d390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f42380c9e50 tx=0x7f4228008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:10.266 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.262+0000 7f4226ffd700 1 -- 172.21.15.91:0/4254739949 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f422c054380 con 0x7f423806b400 2023-11-24T06:08:10.454 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.450+0000 7f423cb30700 1 -- 172.21.15.91:0/4254739949 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4238071e10 con 0x7f421806aed0 2023-11-24T06:08:10.461 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.457+0000 7f4226ffd700 1 -- 172.21.15.91:0/4254739949 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264531 (secure 0 0 0) 0x7f4228009180 con 0x7f421806aed0 2023-11-24T06:08:10.464 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:10.466 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.460+0000 7f423cb30700 1 -- 172.21.15.91:0/4254739949 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f421806aed0 msgr2=0x7f421806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.466 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.460+0000 7f423cb30700 1 --2- 172.21.15.91:0/4254739949 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f421806aed0 0x7f421806d390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f42380c9e50 tx=0x7f4228008040 comp rx=0 tx=0).stop 2023-11-24T06:08:10.466 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.460+0000 7f423cb30700 1 -- 172.21.15.91:0/4254739949 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f423806b400 msgr2=0x7f42380cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.460+0000 7f423cb30700 1 --2- 172.21.15.91:0/4254739949 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f423806b400 0x7f42380cd020 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f422c00c550 tx=0x7f422c00c910 comp rx=0 tx=0).stop 2023-11-24T06:08:10.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.460+0000 7f423cb30700 1 -- 172.21.15.91:0/4254739949 shutdown_connections 2023-11-24T06:08:10.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.460+0000 7f423cb30700 1 --2- 172.21.15.91:0/4254739949 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f421806aed0 0x7f421806d390 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.460+0000 7f423cb30700 1 --2- 172.21.15.91:0/4254739949 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f423815e1e0 0x7f423809b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.460+0000 7f423cb30700 1 --2- 172.21.15.91:0/4254739949 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f423806b400 0x7f42380cd020 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.460+0000 7f423cb30700 1 --2- 172.21.15.91:0/4254739949 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42380b5e40 0x7f42381f3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.460+0000 7f423cb30700 1 -- 172.21.15.91:0/4254739949 >> 172.21.15.91:0/4254739949 conn(0x7f42380b5070 msgr2=0x7f4238160710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:10.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.461+0000 7f423cb30700 1 -- 172.21.15.91:0/4254739949 shutdown_connections 2023-11-24T06:08:10.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.461+0000 7f423cb30700 1 -- 172.21.15.91:0/4254739949 wait complete. 2023-11-24T06:08:10.467 INFO:teuthology.orchestra.run.smithi091.stderr:dumped all 2023-11-24T06:08:10.476 INFO:teuthology.orchestra.run.smithi091.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-11-24T06:08:09.398632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219012,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755708,"statfs":{"total":773094113280,"available":772869844992,"internally_reserved":0,"allocated":2891776,"data_stored":1752836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005345"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859587+0000","last_change":"2023-11-24T06:08:01.435066+0000","last_active":"2023-11-24T06:08:02.859587+0000","last_peered":"2023-11-24T06:08:02.859587+0000","last_clean":"2023-11-24T06:08:02.859587+0000","last_became_active":"2023-11-24T06:08:01.434733+0000","last_became_peered":"2023-11-24T06:08:01.434733+0000","last_unstale":"2023-11-24T06:08:02.859587+0000","last_undegraded":"2023-11-24T06:08:02.859587+0000","last_fullsized":"2023-11-24T06:08:02.859587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:46:08.238325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.428542+0000","last_change":"2023-11-24T06:08:01.428588+0000","last_active":"2023-11-24T06:08:02.428542+0000","last_peered":"2023-11-24T06:08:02.428542+0000","last_clean":"2023-11-24T06:08:02.428542+0000","last_became_active":"2023-11-24T06:08:01.428257+0000","last_became_peered":"2023-11-24T06:08:01.428257+0000","last_unstale":"2023-11-24T06:08:02.428542+0000","last_undegraded":"2023-11-24T06:08:02.428542+0000","last_fullsized":"2023-11-24T06:08:02.428542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:40:20.503614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858341+0000","last_change":"2023-11-24T06:08:01.428960+0000","last_active":"2023-11-24T06:08:02.858341+0000","last_peered":"2023-11-24T06:08:02.858341+0000","last_clean":"2023-11-24T06:08:02.858341+0000","last_became_active":"2023-11-24T06:08:01.428600+0000","last_became_peered":"2023-11-24T06:08:01.428600+0000","last_unstale":"2023-11-24T06:08:02.858341+0000","last_undegraded":"2023-11-24T06:08:02.858341+0000","last_fullsized":"2023-11-24T06:08:02.858341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:52:57.033291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858818+0000","last_change":"2023-11-24T06:08:01.435529+0000","last_active":"2023-11-24T06:08:02.858818+0000","last_peered":"2023-11-24T06:08:02.858818+0000","last_clean":"2023-11-24T06:08:02.858818+0000","last_became_active":"2023-11-24T06:08:01.435128+0000","last_became_peered":"2023-11-24T06:08:01.435128+0000","last_unstale":"2023-11-24T06:08:02.858818+0000","last_undegraded":"2023-11-24T06:08:02.858818+0000","last_fullsized":"2023-11-24T06:08:02.858818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:06:58.673664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859171+0000","last_change":"2023-11-24T06:08:01.434208+0000","last_active":"2023-11-24T06:08:02.859171+0000","last_peered":"2023-11-24T06:08:02.859171+0000","last_clean":"2023-11-24T06:08:02.859171+0000","last_became_active":"2023-11-24T06:08:01.433849+0000","last_became_peered":"2023-11-24T06:08:01.433849+0000","last_unstale":"2023-11-24T06:08:02.859171+0000","last_undegraded":"2023-11-24T06:08:02.859171+0000","last_fullsized":"2023-11-24T06:08:02.859171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:38:12.787229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858655+0000","last_change":"2023-11-24T06:08:01.436361+0000","last_active":"2023-11-24T06:08:02.858655+0000","last_peered":"2023-11-24T06:08:02.858655+0000","last_clean":"2023-11-24T06:08:02.858655+0000","last_became_active":"2023-11-24T06:08:01.434166+0000","last_became_peered":"2023-11-24T06:08:01.434166+0000","last_unstale":"2023-11-24T06:08:02.858655+0000","last_undegraded":"2023-11-24T06:08:02.858655+0000","last_fullsized":"2023-11-24T06:08:02.858655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:22.215190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858814+0000","last_change":"2023-11-24T06:08:01.433541+0000","last_active":"2023-11-24T06:08:02.858814+0000","last_peered":"2023-11-24T06:08:02.858814+0000","last_clean":"2023-11-24T06:08:02.858814+0000","last_became_active":"2023-11-24T06:08:01.433237+0000","last_became_peered":"2023-11-24T06:08:01.433237+0000","last_unstale":"2023-11-24T06:08:02.858814+0000","last_undegraded":"2023-11-24T06:08:02.858814+0000","last_fullsized":"2023-11-24T06:08:02.858814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:32:41.812446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.857978+0000","last_change":"2023-11-24T06:08:01.435678+0000","last_active":"2023-11-24T06:08:02.857978+0000","last_peered":"2023-11-24T06:08:02.857978+0000","last_clean":"2023-11-24T06:08:02.857978+0000","last_became_active":"2023-11-24T06:08:01.435365+0000","last_became_peered":"2023-11-24T06:08:01.435365+0000","last_unstale":"2023-11-24T06:08:02.857978+0000","last_undegraded":"2023-11-24T06:08:02.857978+0000","last_fullsized":"2023-11-24T06:08:02.857978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:55:45.332918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859541+0000","last_change":"2023-11-24T06:08:01.438757+0000","last_active":"2023-11-24T06:08:02.859541+0000","last_peered":"2023-11-24T06:08:02.859541+0000","last_clean":"2023-11-24T06:08:02.859541+0000","last_became_active":"2023-11-24T06:08:01.438495+0000","last_became_peered":"2023-11-24T06:08:01.438495+0000","last_unstale":"2023-11-24T06:08:02.859541+0000","last_undegraded":"2023-11-24T06:08:02.859541+0000","last_fullsized":"2023-11-24T06:08:02.859541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:51:58.808088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.427783+0000","last_change":"2023-11-24T06:08:01.435879+0000","last_active":"2023-11-24T06:08:02.427783+0000","last_peered":"2023-11-24T06:08:02.427783+0000","last_clean":"2023-11-24T06:08:02.427783+0000","last_became_active":"2023-11-24T06:08:01.435556+0000","last_became_peered":"2023-11-24T06:08:01.435556+0000","last_unstale":"2023-11-24T06:08:02.427783+0000","last_undegraded":"2023-11-24T06:08:02.427783+0000","last_fullsized":"2023-11-24T06:08:02.427783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:22.215190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.427548+0000","last_change":"2023-11-24T06:08:01.436002+0000","last_active":"2023-11-24T06:08:02.427548+0000","last_peered":"2023-11-24T06:08:02.427548+0000","last_clean":"2023-11-24T06:08:02.427548+0000","last_became_active":"2023-11-24T06:08:01.435547+0000","last_became_peered":"2023-11-24T06:08:01.435547+0000","last_unstale":"2023-11-24T06:08:02.427548+0000","last_undegraded":"2023-11-24T06:08:02.427548+0000","last_fullsized":"2023-11-24T06:08:02.427548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:28.564398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859683+0000","last_change":"2023-11-24T06:08:01.431781+0000","last_active":"2023-11-24T06:08:02.859683+0000","last_peered":"2023-11-24T06:08:02.859683+0000","last_clean":"2023-11-24T06:08:02.859683+0000","last_became_active":"2023-11-24T06:08:01.431494+0000","last_became_peered":"2023-11-24T06:08:01.431494+0000","last_unstale":"2023-11-24T06:08:02.859683+0000","last_undegraded":"2023-11-24T06:08:02.859683+0000","last_fullsized":"2023-11-24T06:08:02.859683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:37:39.086654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.857833+0000","last_change":"2023-11-24T06:08:01.434455+0000","last_active":"2023-11-24T06:08:02.857833+0000","last_peered":"2023-11-24T06:08:02.857833+0000","last_clean":"2023-11-24T06:08:02.857833+0000","last_became_active":"2023-11-24T06:08:01.434201+0000","last_became_peered":"2023-11-24T06:08:01.434201+0000","last_unstale":"2023-11-24T06:08:02.857833+0000","last_undegraded":"2023-11-24T06:08:02.857833+0000","last_fullsized":"2023-11-24T06:08:02.857833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:52:57.033291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.857927+0000","last_change":"2023-11-24T06:08:01.427810+0000","last_active":"2023-11-24T06:08:02.857927+0000","last_peered":"2023-11-24T06:08:02.857927+0000","last_clean":"2023-11-24T06:08:02.857927+0000","last_became_active":"2023-11-24T06:08:01.427545+0000","last_became_peered":"2023-11-24T06:08:01.427545+0000","last_unstale":"2023-11-24T06:08:02.857927+0000","last_undegraded":"2023-11-24T06:08:02.857927+0000","last_fullsized":"2023-11-24T06:08:02.857927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:30.200581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.428420+0000","last_change":"2023-11-24T06:08:01.431162+0000","last_active":"2023-11-24T06:08:02.428420+0000","last_peered":"2023-11-24T06:08:02.428420+0000","last_clean":"2023-11-24T06:08:02.428420+0000","last_became_active":"2023-11-24T06:08:01.430862+0000","last_became_peered":"2023-11-24T06:08:01.430862+0000","last_unstale":"2023-11-24T06:08:02.428420+0000","last_undegraded":"2023-11-24T06:08:02.428420+0000","last_fullsized":"2023-11-24T06:08:02.428420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:52:57.033291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858252+0000","last_change":"2023-11-24T06:08:01.434078+0000","last_active":"2023-11-24T06:08:02.858252+0000","last_peered":"2023-11-24T06:08:02.858252+0000","last_clean":"2023-11-24T06:08:02.858252+0000","last_became_active":"2023-11-24T06:08:01.433636+0000","last_became_peered":"2023-11-24T06:08:01.433636+0000","last_unstale":"2023-11-24T06:08:02.858252+0000","last_undegraded":"2023-11-24T06:08:02.858252+0000","last_fullsized":"2023-11-24T06:08:02.858252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:47:16.508162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859546+0000","last_change":"2023-11-24T06:08:01.436588+0000","last_active":"2023-11-24T06:08:02.859546+0000","last_peered":"2023-11-24T06:08:02.859546+0000","last_clean":"2023-11-24T06:08:02.859546+0000","last_became_active":"2023-11-24T06:08:01.436193+0000","last_became_peered":"2023-11-24T06:08:01.436193+0000","last_unstale":"2023-11-24T06:08:02.859546+0000","last_undegraded":"2023-11-24T06:08:02.859546+0000","last_fullsized":"2023-11-24T06:08:02.859546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:51:58.808088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.429338+0000","last_change":"2023-11-24T06:08:01.424485+0000","last_active":"2023-11-24T06:08:02.429338+0000","last_peered":"2023-11-24T06:08:02.429338+0000","last_clean":"2023-11-24T06:08:02.429338+0000","last_became_active":"2023-11-24T06:08:01.424234+0000","last_became_peered":"2023-11-24T06:08:01.424234+0000","last_unstale":"2023-11-24T06:08:02.429338+0000","last_undegraded":"2023-11-24T06:08:02.429338+0000","last_fullsized":"2023-11-24T06:08:02.429338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:29.649298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859691+0000","last_change":"2023-11-24T06:08:01.434319+0000","last_active":"2023-11-24T06:08:02.859691+0000","last_peered":"2023-11-24T06:08:02.859691+0000","last_clean":"2023-11-24T06:08:02.859691+0000","last_became_active":"2023-11-24T06:08:01.433984+0000","last_became_peered":"2023-11-24T06:08:01.433984+0000","last_unstale":"2023-11-24T06:08:02.859691+0000","last_undegraded":"2023-11-24T06:08:02.859691+0000","last_fullsized":"2023-11-24T06:08:02.859691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:30:33.401914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.407937+0000","last_change":"2023-11-24T06:08:00.418990+0000","last_active":"2023-11-24T06:08:02.407937+0000","last_peered":"2023-11-24T06:08:02.407937+0000","last_clean":"2023-11-24T06:08:02.407937+0000","last_became_active":"2023-11-24T06:08:00.418683+0000","last_became_peered":"2023-11-24T06:08:00.418683+0000","last_unstale":"2023-11-24T06:08:02.407937+0000","last_undegraded":"2023-11-24T06:08:02.407937+0000","last_fullsized":"2023-11-24T06:08:02.407937+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:13.152063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858033+0000","last_change":"2023-11-24T06:08:01.431913+0000","last_active":"2023-11-24T06:08:02.858033+0000","last_peered":"2023-11-24T06:08:02.858033+0000","last_clean":"2023-11-24T06:08:02.858033+0000","last_became_active":"2023-11-24T06:08:01.431373+0000","last_became_peered":"2023-11-24T06:08:01.431373+0000","last_unstale":"2023-11-24T06:08:02.858033+0000","last_undegraded":"2023-11-24T06:08:02.858033+0000","last_fullsized":"2023-11-24T06:08:02.858033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:28.564398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.406361+0000","last_change":"2023-11-24T06:08:00.424674+0000","last_active":"2023-11-24T06:08:02.406361+0000","last_peered":"2023-11-24T06:08:02.406361+0000","last_clean":"2023-11-24T06:08:02.406361+0000","last_became_active":"2023-11-24T06:08:00.424282+0000","last_became_peered":"2023-11-24T06:08:00.424282+0000","last_unstale":"2023-11-24T06:08:02.406361+0000","last_undegraded":"2023-11-24T06:08:02.406361+0000","last_fullsized":"2023-11-24T06:08:02.406361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:03.380514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858392+0000","last_change":"2023-11-24T06:08:01.434659+0000","last_active":"2023-11-24T06:08:02.858392+0000","last_peered":"2023-11-24T06:08:02.858392+0000","last_clean":"2023-11-24T06:08:02.858392+0000","last_became_active":"2023-11-24T06:08:01.434373+0000","last_became_peered":"2023-11-24T06:08:01.434373+0000","last_unstale":"2023-11-24T06:08:02.858392+0000","last_undegraded":"2023-11-24T06:08:02.858392+0000","last_fullsized":"2023-11-24T06:08:02.858392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:40:20.503614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.405297+0000","last_change":"2023-11-24T06:08:00.426586+0000","last_active":"2023-11-24T06:08:02.405297+0000","last_peered":"2023-11-24T06:08:02.405297+0000","last_clean":"2023-11-24T06:08:02.405297+0000","last_became_active":"2023-11-24T06:08:00.426279+0000","last_became_peered":"2023-11-24T06:08:00.426279+0000","last_unstale":"2023-11-24T06:08:02.405297+0000","last_undegraded":"2023-11-24T06:08:02.405297+0000","last_fullsized":"2023-11-24T06:08:02.405297+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:06:57.668726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858648+0000","last_change":"2023-11-24T06:08:01.439887+0000","last_active":"2023-11-24T06:08:02.858648+0000","last_peered":"2023-11-24T06:08:02.858648+0000","last_clean":"2023-11-24T06:08:02.858648+0000","last_became_active":"2023-11-24T06:08:01.439606+0000","last_became_peered":"2023-11-24T06:08:01.439606+0000","last_unstale":"2023-11-24T06:08:02.858648+0000","last_undegraded":"2023-11-24T06:08:02.858648+0000","last_fullsized":"2023-11-24T06:08:02.858648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:16:08.901829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.405228+0000","last_change":"2023-11-24T06:08:00.423364+0000","last_active":"2023-11-24T06:08:02.405228+0000","last_peered":"2023-11-24T06:08:02.405228+0000","last_clean":"2023-11-24T06:08:02.405228+0000","last_became_active":"2023-11-24T06:08:00.423062+0000","last_became_peered":"2023-11-24T06:08:00.423062+0000","last_unstale":"2023-11-24T06:08:02.405228+0000","last_undegraded":"2023-11-24T06:08:02.405228+0000","last_fullsized":"2023-11-24T06:08:02.405228+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:28.644360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859170+0000","last_change":"2023-11-24T06:08:01.438369+0000","last_active":"2023-11-24T06:08:02.859170+0000","last_peered":"2023-11-24T06:08:02.859170+0000","last_clean":"2023-11-24T06:08:02.859170+0000","last_became_active":"2023-11-24T06:08:01.438112+0000","last_became_peered":"2023-11-24T06:08:01.438112+0000","last_unstale":"2023-11-24T06:08:02.859170+0000","last_undegraded":"2023-11-24T06:08:02.859170+0000","last_fullsized":"2023-11-24T06:08:02.859170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:02:53.563916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.406278+0000","last_change":"2023-11-24T06:08:00.425844+0000","last_active":"2023-11-24T06:08:02.406278+0000","last_peered":"2023-11-24T06:08:02.406278+0000","last_clean":"2023-11-24T06:08:02.406278+0000","last_became_active":"2023-11-24T06:08:00.425587+0000","last_became_peered":"2023-11-24T06:08:00.425587+0000","last_unstale":"2023-11-24T06:08:02.406278+0000","last_undegraded":"2023-11-24T06:08:02.406278+0000","last_fullsized":"2023-11-24T06:08:02.406278+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:29.195643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858441+0000","last_change":"2023-11-24T06:08:01.427389+0000","last_active":"2023-11-24T06:08:02.858441+0000","last_peered":"2023-11-24T06:08:02.858441+0000","last_clean":"2023-11-24T06:08:02.858441+0000","last_became_active":"2023-11-24T06:08:01.427038+0000","last_became_peered":"2023-11-24T06:08:01.427038+0000","last_unstale":"2023-11-24T06:08:02.858441+0000","last_undegraded":"2023-11-24T06:08:02.858441+0000","last_fullsized":"2023-11-24T06:08:02.858441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:32.299407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.402091+0000","last_change":"2023-11-24T06:08:00.411792+0000","last_active":"2023-11-24T06:08:02.402091+0000","last_peered":"2023-11-24T06:08:02.402091+0000","last_clean":"2023-11-24T06:08:02.402091+0000","last_became_active":"2023-11-24T06:08:00.411486+0000","last_became_peered":"2023-11-24T06:08:00.411486+0000","last_unstale":"2023-11-24T06:08:02.402091+0000","last_undegraded":"2023-11-24T06:08:02.402091+0000","last_fullsized":"2023-11-24T06:08:02.402091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:41.551384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858551+0000","last_change":"2023-11-24T06:08:01.438231+0000","last_active":"2023-11-24T06:08:02.858551+0000","last_peered":"2023-11-24T06:08:02.858551+0000","last_clean":"2023-11-24T06:08:02.858551+0000","last_became_active":"2023-11-24T06:08:01.437944+0000","last_became_peered":"2023-11-24T06:08:01.437944+0000","last_unstale":"2023-11-24T06:08:02.858551+0000","last_undegraded":"2023-11-24T06:08:02.858551+0000","last_fullsized":"2023-11-24T06:08:02.858551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:22.215190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.406223+0000","last_change":"2023-11-24T06:08:00.416967+0000","last_active":"2023-11-24T06:08:02.406223+0000","last_peered":"2023-11-24T06:08:02.406223+0000","last_clean":"2023-11-24T06:08:02.406223+0000","last_became_active":"2023-11-24T06:08:00.416532+0000","last_became_peered":"2023-11-24T06:08:00.416532+0000","last_unstale":"2023-11-24T06:08:02.406223+0000","last_undegraded":"2023-11-24T06:08:02.406223+0000","last_fullsized":"2023-11-24T06:08:02.406223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:20.079860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858392+0000","last_change":"2023-11-24T06:08:01.429802+0000","last_active":"2023-11-24T06:08:02.858392+0000","last_peered":"2023-11-24T06:08:02.858392+0000","last_clean":"2023-11-24T06:08:02.858392+0000","last_became_active":"2023-11-24T06:08:01.429578+0000","last_became_peered":"2023-11-24T06:08:01.429578+0000","last_unstale":"2023-11-24T06:08:02.858392+0000","last_undegraded":"2023-11-24T06:08:02.858392+0000","last_fullsized":"2023-11-24T06:08:02.858392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:32:41.812446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.401833+0000","last_change":"2023-11-24T06:08:00.422030+0000","last_active":"2023-11-24T06:08:02.401833+0000","last_peered":"2023-11-24T06:08:02.401833+0000","last_clean":"2023-11-24T06:08:02.401833+0000","last_became_active":"2023-11-24T06:08:00.421740+0000","last_became_peered":"2023-11-24T06:08:00.421740+0000","last_unstale":"2023-11-24T06:08:02.401833+0000","last_undegraded":"2023-11-24T06:08:02.401833+0000","last_fullsized":"2023-11-24T06:08:02.401833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:58.808054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.428563+0000","last_change":"2023-11-24T06:08:01.433496+0000","last_active":"2023-11-24T06:08:02.428563+0000","last_peered":"2023-11-24T06:08:02.428563+0000","last_clean":"2023-11-24T06:08:02.428563+0000","last_became_active":"2023-11-24T06:08:01.433238+0000","last_became_peered":"2023-11-24T06:08:01.433238+0000","last_unstale":"2023-11-24T06:08:02.428563+0000","last_undegraded":"2023-11-24T06:08:02.428563+0000","last_fullsized":"2023-11-24T06:08:02.428563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:28.564398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.402192+0000","last_change":"2023-11-24T06:08:00.422595+0000","last_active":"2023-11-24T06:08:02.402192+0000","last_peered":"2023-11-24T06:08:02.402192+0000","last_clean":"2023-11-24T06:08:02.402192+0000","last_became_active":"2023-11-24T06:08:00.422365+0000","last_became_peered":"2023-11-24T06:08:00.422365+0000","last_unstale":"2023-11-24T06:08:02.402192+0000","last_undegraded":"2023-11-24T06:08:02.402192+0000","last_fullsized":"2023-11-24T06:08:02.402192+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:39.360506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859469+0000","last_change":"2023-11-24T06:08:01.434890+0000","last_active":"2023-11-24T06:08:02.859469+0000","last_peered":"2023-11-24T06:08:02.859469+0000","last_clean":"2023-11-24T06:08:02.859469+0000","last_became_active":"2023-11-24T06:08:01.434706+0000","last_became_peered":"2023-11-24T06:08:01.434706+0000","last_unstale":"2023-11-24T06:08:02.859469+0000","last_undegraded":"2023-11-24T06:08:02.859469+0000","last_fullsized":"2023-11-24T06:08:02.859469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:19:13.981084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.402050+0000","last_change":"2023-11-24T06:08:00.423297+0000","last_active":"2023-11-24T06:08:02.402050+0000","last_peered":"2023-11-24T06:08:02.402050+0000","last_clean":"2023-11-24T06:08:02.402050+0000","last_became_active":"2023-11-24T06:08:00.423028+0000","last_became_peered":"2023-11-24T06:08:00.423028+0000","last_unstale":"2023-11-24T06:08:02.402050+0000","last_undegraded":"2023-11-24T06:08:02.402050+0000","last_fullsized":"2023-11-24T06:08:02.402050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:37:38.081716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858516+0000","last_change":"2023-11-24T06:08:01.437725+0000","last_active":"2023-11-24T06:08:02.858516+0000","last_peered":"2023-11-24T06:08:02.858516+0000","last_clean":"2023-11-24T06:08:02.858516+0000","last_became_active":"2023-11-24T06:08:01.437464+0000","last_became_peered":"2023-11-24T06:08:01.437464+0000","last_unstale":"2023-11-24T06:08:02.858516+0000","last_undegraded":"2023-11-24T06:08:02.858516+0000","last_fullsized":"2023-11-24T06:08:02.858516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:52.422428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.404823+0000","last_change":"2023-11-24T06:08:00.421041+0000","last_active":"2023-11-24T06:08:02.404823+0000","last_peered":"2023-11-24T06:08:02.404823+0000","last_clean":"2023-11-24T06:08:02.404823+0000","last_became_active":"2023-11-24T06:08:00.420677+0000","last_became_peered":"2023-11-24T06:08:00.420677+0000","last_unstale":"2023-11-24T06:08:02.404823+0000","last_undegraded":"2023-11-24T06:08:02.404823+0000","last_fullsized":"2023-11-24T06:08:02.404823+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:29.195643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859100+0000","last_change":"2023-11-24T06:08:01.437971+0000","last_active":"2023-11-24T06:08:02.859100+0000","last_peered":"2023-11-24T06:08:02.859100+0000","last_clean":"2023-11-24T06:08:02.859100+0000","last_became_active":"2023-11-24T06:08:01.437640+0000","last_became_peered":"2023-11-24T06:08:01.437640+0000","last_unstale":"2023-11-24T06:08:02.859100+0000","last_undegraded":"2023-11-24T06:08:02.859100+0000","last_fullsized":"2023-11-24T06:08:02.859100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:16.537475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.402105+0000","last_change":"2023-11-24T06:08:00.418490+0000","last_active":"2023-11-24T06:08:02.402105+0000","last_peered":"2023-11-24T06:08:02.402105+0000","last_clean":"2023-11-24T06:08:02.402105+0000","last_became_active":"2023-11-24T06:08:00.418200+0000","last_became_peered":"2023-11-24T06:08:00.418200+0000","last_unstale":"2023-11-24T06:08:02.402105+0000","last_undegraded":"2023-11-24T06:08:02.402105+0000","last_fullsized":"2023-11-24T06:08:02.402105+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:03.380514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859707+0000","last_change":"2023-11-24T06:08:01.439461+0000","last_active":"2023-11-24T06:08:02.859707+0000","last_peered":"2023-11-24T06:08:02.859707+0000","last_clean":"2023-11-24T06:08:02.859707+0000","last_became_active":"2023-11-24T06:08:01.439210+0000","last_became_peered":"2023-11-24T06:08:01.439210+0000","last_unstale":"2023-11-24T06:08:02.859707+0000","last_undegraded":"2023-11-24T06:08:02.859707+0000","last_fullsized":"2023-11-24T06:08:02.859707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:49:31.031906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.407837+0000","last_change":"2023-11-24T06:08:00.423172+0000","last_active":"2023-11-24T06:08:02.407837+0000","last_peered":"2023-11-24T06:08:02.407837+0000","last_clean":"2023-11-24T06:08:02.407837+0000","last_became_active":"2023-11-24T06:08:00.422899+0000","last_became_peered":"2023-11-24T06:08:00.422899+0000","last_unstale":"2023-11-24T06:08:02.407837+0000","last_undegraded":"2023-11-24T06:08:02.407837+0000","last_fullsized":"2023-11-24T06:08:02.407837+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:17.143751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.857787+0000","last_change":"2023-11-24T06:08:01.435150+0000","last_active":"2023-11-24T06:08:02.857787+0000","last_peered":"2023-11-24T06:08:02.857787+0000","last_clean":"2023-11-24T06:08:02.857787+0000","last_became_active":"2023-11-24T06:08:01.434861+0000","last_became_peered":"2023-11-24T06:08:01.434861+0000","last_unstale":"2023-11-24T06:08:02.857787+0000","last_undegraded":"2023-11-24T06:08:02.857787+0000","last_fullsized":"2023-11-24T06:08:02.857787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:40.365444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.401913+0000","last_change":"2023-11-24T06:08:00.422660+0000","last_active":"2023-11-24T06:08:02.401913+0000","last_peered":"2023-11-24T06:08:02.401913+0000","last_clean":"2023-11-24T06:08:02.401913+0000","last_became_active":"2023-11-24T06:08:00.421922+0000","last_became_peered":"2023-11-24T06:08:00.421922+0000","last_unstale":"2023-11-24T06:08:02.401913+0000","last_undegraded":"2023-11-24T06:08:02.401913+0000","last_fullsized":"2023-11-24T06:08:02.401913+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:27.559460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859321+0000","last_change":"2023-11-24T06:08:01.442581+0000","last_active":"2023-11-24T06:08:02.859321+0000","last_peered":"2023-11-24T06:08:02.859321+0000","last_clean":"2023-11-24T06:08:02.859321+0000","last_became_active":"2023-11-24T06:08:01.442145+0000","last_became_peered":"2023-11-24T06:08:01.442145+0000","last_unstale":"2023-11-24T06:08:02.859321+0000","last_undegraded":"2023-11-24T06:08:02.859321+0000","last_fullsized":"2023-11-24T06:08:02.859321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:16:08.901829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.402064+0000","last_change":"2023-11-24T06:08:00.413903+0000","last_active":"2023-11-24T06:08:02.402064+0000","last_peered":"2023-11-24T06:08:02.402064+0000","last_clean":"2023-11-24T06:08:02.402064+0000","last_became_active":"2023-11-24T06:08:00.413581+0000","last_became_peered":"2023-11-24T06:08:00.413581+0000","last_unstale":"2023-11-24T06:08:02.402064+0000","last_undegraded":"2023-11-24T06:08:02.402064+0000","last_fullsized":"2023-11-24T06:08:02.402064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:15.532537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.427222+0000","last_change":"2023-11-24T06:08:01.429649+0000","last_active":"2023-11-24T06:08:02.427222+0000","last_peered":"2023-11-24T06:08:02.427222+0000","last_clean":"2023-11-24T06:08:02.427222+0000","last_became_active":"2023-11-24T06:08:01.429189+0000","last_became_peered":"2023-11-24T06:08:01.429189+0000","last_unstale":"2023-11-24T06:08:02.427222+0000","last_undegraded":"2023-11-24T06:08:02.427222+0000","last_fullsized":"2023-11-24T06:08:02.427222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:59.812992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.407695+0000","last_change":"2023-11-24T06:08:00.417220+0000","last_active":"2023-11-24T06:08:02.407695+0000","last_peered":"2023-11-24T06:08:02.407695+0000","last_clean":"2023-11-24T06:08:02.407695+0000","last_became_active":"2023-11-24T06:08:00.416947+0000","last_became_peered":"2023-11-24T06:08:00.416947+0000","last_unstale":"2023-11-24T06:08:02.407695+0000","last_undegraded":"2023-11-24T06:08:02.407695+0000","last_fullsized":"2023-11-24T06:08:02.407695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:18.816306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859012+0000","last_change":"2023-11-24T06:08:01.442182+0000","last_active":"2023-11-24T06:08:02.859012+0000","last_peered":"2023-11-24T06:08:02.859012+0000","last_clean":"2023-11-24T06:08:02.859012+0000","last_became_active":"2023-11-24T06:08:01.441895+0000","last_became_peered":"2023-11-24T06:08:01.441895+0000","last_unstale":"2023-11-24T06:08:02.859012+0000","last_undegraded":"2023-11-24T06:08:02.859012+0000","last_fullsized":"2023-11-24T06:08:02.859012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:55:45.332918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.405064+0000","last_change":"2023-11-24T06:08:00.420431+0000","last_active":"2023-11-24T06:08:02.405064+0000","last_peered":"2023-11-24T06:08:02.405064+0000","last_clean":"2023-11-24T06:08:02.405064+0000","last_became_active":"2023-11-24T06:08:00.420108+0000","last_became_peered":"2023-11-24T06:08:00.420108+0000","last_unstale":"2023-11-24T06:08:02.405064+0000","last_undegraded":"2023-11-24T06:08:02.405064+0000","last_fullsized":"2023-11-24T06:08:02.405064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:11.429196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858885+0000","last_change":"2023-11-24T06:08:01.441968+0000","last_active":"2023-11-24T06:08:02.858885+0000","last_peered":"2023-11-24T06:08:02.858885+0000","last_clean":"2023-11-24T06:08:02.858885+0000","last_became_active":"2023-11-24T06:08:01.440920+0000","last_became_peered":"2023-11-24T06:08:01.440920+0000","last_unstale":"2023-11-24T06:08:02.858885+0000","last_undegraded":"2023-11-24T06:08:02.858885+0000","last_fullsized":"2023-11-24T06:08:02.858885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:02:53.563916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.404769+0000","last_change":"2023-11-24T06:08:00.423440+0000","last_active":"2023-11-24T06:08:02.404769+0000","last_peered":"2023-11-24T06:08:02.404769+0000","last_clean":"2023-11-24T06:08:02.404769+0000","last_became_active":"2023-11-24T06:08:00.423175+0000","last_became_peered":"2023-11-24T06:08:00.423175+0000","last_unstale":"2023-11-24T06:08:02.404769+0000","last_undegraded":"2023-11-24T06:08:02.404769+0000","last_fullsized":"2023-11-24T06:08:02.404769+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:39.360506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859329+0000","last_change":"2023-11-24T06:08:01.435280+0000","last_active":"2023-11-24T06:08:02.859329+0000","last_peered":"2023-11-24T06:08:02.859329+0000","last_clean":"2023-11-24T06:08:02.859329+0000","last_became_active":"2023-11-24T06:08:01.434963+0000","last_became_peered":"2023-11-24T06:08:01.434963+0000","last_unstale":"2023-11-24T06:08:02.859329+0000","last_undegraded":"2023-11-24T06:08:02.859329+0000","last_fullsized":"2023-11-24T06:08:02.859329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:38:12.787229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.401899+0000","last_change":"2023-11-24T06:08:00.420129+0000","last_active":"2023-11-24T06:08:02.401899+0000","last_peered":"2023-11-24T06:08:02.401899+0000","last_clean":"2023-11-24T06:08:02.401899+0000","last_became_active":"2023-11-24T06:08:00.418612+0000","last_became_peered":"2023-11-24T06:08:00.418612+0000","last_unstale":"2023-11-24T06:08:02.401899+0000","last_undegraded":"2023-11-24T06:08:02.401899+0000","last_fullsized":"2023-11-24T06:08:02.401899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:40:19.498676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859549+0000","last_change":"2023-11-24T06:08:01.430171+0000","last_active":"2023-11-24T06:08:02.859549+0000","last_peered":"2023-11-24T06:08:02.859549+0000","last_clean":"2023-11-24T06:08:02.859549+0000","last_became_active":"2023-11-24T06:08:01.428562+0000","last_became_peered":"2023-11-24T06:08:01.428562+0000","last_unstale":"2023-11-24T06:08:02.859549+0000","last_undegraded":"2023-11-24T06:08:02.859549+0000","last_fullsized":"2023-11-24T06:08:02.859549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:29.649298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.428323+0000","last_change":"2023-11-24T06:08:01.433086+0000","last_active":"2023-11-24T06:08:02.428323+0000","last_peered":"2023-11-24T06:08:02.428323+0000","last_clean":"2023-11-24T06:08:02.428323+0000","last_became_active":"2023-11-24T06:08:01.432747+0000","last_became_peered":"2023-11-24T06:08:01.432747+0000","last_unstale":"2023-11-24T06:08:02.428323+0000","last_undegraded":"2023-11-24T06:08:02.428323+0000","last_fullsized":"2023-11-24T06:08:02.428323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:59.812992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.407725+0000","last_change":"2023-11-24T06:08:00.416138+0000","last_active":"2023-11-24T06:08:02.407725+0000","last_peered":"2023-11-24T06:08:02.407725+0000","last_clean":"2023-11-24T06:08:02.407725+0000","last_became_active":"2023-11-24T06:08:00.415909+0000","last_became_peered":"2023-11-24T06:08:00.415909+0000","last_unstale":"2023-11-24T06:08:02.407725+0000","last_undegraded":"2023-11-24T06:08:02.407725+0000","last_fullsized":"2023-11-24T06:08:02.407725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:31.619293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858173+0000","last_change":"2023-11-24T06:08:01.438377+0000","last_active":"2023-11-24T06:08:02.858173+0000","last_peered":"2023-11-24T06:08:02.858173+0000","last_clean":"2023-11-24T06:08:02.858173+0000","last_became_active":"2023-11-24T06:08:01.438142+0000","last_became_peered":"2023-11-24T06:08:01.438142+0000","last_unstale":"2023-11-24T06:08:02.858173+0000","last_undegraded":"2023-11-24T06:08:02.858173+0000","last_fullsized":"2023-11-24T06:08:02.858173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:22.215190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.401927+0000","last_change":"2023-11-24T06:08:00.419708+0000","last_active":"2023-11-24T06:08:02.401927+0000","last_peered":"2023-11-24T06:08:02.401927+0000","last_clean":"2023-11-24T06:08:02.401927+0000","last_became_active":"2023-11-24T06:08:00.419444+0000","last_became_peered":"2023-11-24T06:08:00.419444+0000","last_unstale":"2023-11-24T06:08:02.401927+0000","last_undegraded":"2023-11-24T06:08:02.401927+0000","last_fullsized":"2023-11-24T06:08:02.401927+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:13.152063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425349+0000","last_change":"2023-11-24T06:07:49.463719+0000","last_active":"2023-11-24T06:08:01.425349+0000","last_peered":"2023-11-24T06:08:01.425349+0000","last_clean":"2023-11-24T06:08:01.425349+0000","last_became_active":"2023-11-24T06:07:49.462736+0000","last_became_peered":"2023-11-24T06:07:49.462736+0000","last_unstale":"2023-11-24T06:08:01.425349+0000","last_undegraded":"2023-11-24T06:08:01.425349+0000","last_fullsized":"2023-11-24T06:08:01.425349+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_clean_scrub_stamp":"2023-11-24T06:07:47.860604+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:16.508770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.857710+0000","last_change":"2023-11-24T06:08:01.438231+0000","last_active":"2023-11-24T06:08:02.857710+0000","last_peered":"2023-11-24T06:08:02.857710+0000","last_clean":"2023-11-24T06:08:02.857710+0000","last_became_active":"2023-11-24T06:08:01.437970+0000","last_became_peered":"2023-11-24T06:08:01.437970+0000","last_unstale":"2023-11-24T06:08:02.857710+0000","last_undegraded":"2023-11-24T06:08:02.857710+0000","last_fullsized":"2023-11-24T06:08:02.857710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:04.385452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.405157+0000","last_change":"2023-11-24T06:08:00.423316+0000","last_active":"2023-11-24T06:08:02.405157+0000","last_peered":"2023-11-24T06:08:02.405157+0000","last_clean":"2023-11-24T06:08:02.405157+0000","last_became_active":"2023-11-24T06:08:00.423046+0000","last_became_peered":"2023-11-24T06:08:00.423046+0000","last_unstale":"2023-11-24T06:08:02.405157+0000","last_undegraded":"2023-11-24T06:08:02.405157+0000","last_fullsized":"2023-11-24T06:08:02.405157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:17.143751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.405285+0000","last_change":"2023-11-24T06:08:00.424803+0000","last_active":"2023-11-24T06:08:02.405285+0000","last_peered":"2023-11-24T06:08:02.405285+0000","last_clean":"2023-11-24T06:08:02.405285+0000","last_became_active":"2023-11-24T06:08:00.424473+0000","last_became_peered":"2023-11-24T06:08:00.424473+0000","last_unstale":"2023-11-24T06:08:02.405285+0000","last_undegraded":"2023-11-24T06:08:02.405285+0000","last_fullsized":"2023-11-24T06:08:02.405285+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:00.828006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.428618+0000","last_change":"2023-11-24T06:08:01.432093+0000","last_active":"2023-11-24T06:08:02.428618+0000","last_peered":"2023-11-24T06:08:02.428618+0000","last_clean":"2023-11-24T06:08:02.428618+0000","last_became_active":"2023-11-24T06:08:01.431795+0000","last_became_peered":"2023-11-24T06:08:01.431795+0000","last_unstale":"2023-11-24T06:08:02.428618+0000","last_undegraded":"2023-11-24T06:08:02.428618+0000","last_fullsized":"2023-11-24T06:08:02.428618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:47:16.508162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.428443+0000","last_change":"2023-11-24T06:08:00.416062+0000","last_active":"2023-11-24T06:08:02.428443+0000","last_peered":"2023-11-24T06:08:02.428443+0000","last_clean":"2023-11-24T06:08:02.428443+0000","last_became_active":"2023-11-24T06:08:00.415783+0000","last_became_peered":"2023-11-24T06:08:00.415783+0000","last_unstale":"2023-11-24T06:08:02.428443+0000","last_undegraded":"2023-11-24T06:08:02.428443+0000","last_fullsized":"2023-11-24T06:08:02.428443+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:03.380514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858401+0000","last_change":"2023-11-24T06:08:01.437911+0000","last_active":"2023-11-24T06:08:02.858401+0000","last_peered":"2023-11-24T06:08:02.858401+0000","last_clean":"2023-11-24T06:08:02.858401+0000","last_became_active":"2023-11-24T06:08:01.437544+0000","last_became_peered":"2023-11-24T06:08:01.437544+0000","last_unstale":"2023-11-24T06:08:02.858401+0000","last_undegraded":"2023-11-24T06:08:02.858401+0000","last_fullsized":"2023-11-24T06:08:02.858401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:02:53.563916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.402168+0000","last_change":"2023-11-24T06:08:00.421447+0000","last_active":"2023-11-24T06:08:02.402168+0000","last_peered":"2023-11-24T06:08:02.402168+0000","last_clean":"2023-11-24T06:08:02.402168+0000","last_became_active":"2023-11-24T06:08:00.421145+0000","last_became_peered":"2023-11-24T06:08:00.421145+0000","last_unstale":"2023-11-24T06:08:02.402168+0000","last_undegraded":"2023-11-24T06:08:02.402168+0000","last_fullsized":"2023-11-24T06:08:02.402168+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:29.195643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858415+0000","last_change":"2023-11-24T06:08:01.427774+0000","last_active":"2023-11-24T06:08:02.858415+0000","last_peered":"2023-11-24T06:08:02.858415+0000","last_clean":"2023-11-24T06:08:02.858415+0000","last_became_active":"2023-11-24T06:08:01.427435+0000","last_became_peered":"2023-11-24T06:08:01.427435+0000","last_unstale":"2023-11-24T06:08:02.858415+0000","last_undegraded":"2023-11-24T06:08:02.858415+0000","last_fullsized":"2023-11-24T06:08:02.858415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:28.564398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.405130+0000","last_change":"2023-11-24T06:08:00.424213+0000","last_active":"2023-11-24T06:08:02.405130+0000","last_peered":"2023-11-24T06:08:02.405130+0000","last_clean":"2023-11-24T06:08:02.405130+0000","last_became_active":"2023-11-24T06:08:00.423943+0000","last_became_peered":"2023-11-24T06:08:00.423943+0000","last_unstale":"2023-11-24T06:08:02.405130+0000","last_undegraded":"2023-11-24T06:08:02.405130+0000","last_fullsized":"2023-11-24T06:08:02.405130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:15.532537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858798+0000","last_change":"2023-11-24T06:08:01.420581+0000","last_active":"2023-11-24T06:08:02.858798+0000","last_peered":"2023-11-24T06:08:02.858798+0000","last_clean":"2023-11-24T06:08:02.858798+0000","last_became_active":"2023-11-24T06:08:01.420103+0000","last_became_peered":"2023-11-24T06:08:01.420103+0000","last_unstale":"2023-11-24T06:08:02.858798+0000","last_undegraded":"2023-11-24T06:08:02.858798+0000","last_fullsized":"2023-11-24T06:08:02.858798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:42.556322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.404691+0000","last_change":"2023-11-24T06:08:00.420593+0000","last_active":"2023-11-24T06:08:02.404691+0000","last_peered":"2023-11-24T06:08:02.404691+0000","last_clean":"2023-11-24T06:08:02.404691+0000","last_became_active":"2023-11-24T06:08:00.420286+0000","last_became_peered":"2023-11-24T06:08:00.420286+0000","last_unstale":"2023-11-24T06:08:02.404691+0000","last_undegraded":"2023-11-24T06:08:02.404691+0000","last_fullsized":"2023-11-24T06:08:02.404691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:28.644360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858190+0000","last_change":"2023-11-24T06:08:01.427055+0000","last_active":"2023-11-24T06:08:02.858190+0000","last_peered":"2023-11-24T06:08:02.858190+0000","last_clean":"2023-11-24T06:08:02.858190+0000","last_became_active":"2023-11-24T06:08:01.426823+0000","last_became_peered":"2023-11-24T06:08:01.426823+0000","last_unstale":"2023-11-24T06:08:02.858190+0000","last_undegraded":"2023-11-24T06:08:02.858190+0000","last_fullsized":"2023-11-24T06:08:02.858190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:06:58.673664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.404734+0000","last_change":"2023-11-24T06:08:00.421597+0000","last_active":"2023-11-24T06:08:02.404734+0000","last_peered":"2023-11-24T06:08:02.404734+0000","last_clean":"2023-11-24T06:08:02.404734+0000","last_became_active":"2023-11-24T06:08:00.421316+0000","last_became_peered":"2023-11-24T06:08:00.421316+0000","last_unstale":"2023-11-24T06:08:02.404734+0000","last_undegraded":"2023-11-24T06:08:02.404734+0000","last_fullsized":"2023-11-24T06:08:02.404734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:17.143751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858069+0000","last_change":"2023-11-24T06:08:01.430666+0000","last_active":"2023-11-24T06:08:02.858069+0000","last_peered":"2023-11-24T06:08:02.858069+0000","last_clean":"2023-11-24T06:08:02.858069+0000","last_became_active":"2023-11-24T06:08:01.430401+0000","last_became_peered":"2023-11-24T06:08:01.430401+0000","last_unstale":"2023-11-24T06:08:02.858069+0000","last_undegraded":"2023-11-24T06:08:02.858069+0000","last_fullsized":"2023-11-24T06:08:02.858069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:47:16.508162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.405217+0000","last_change":"2023-11-24T06:08:00.424291+0000","last_active":"2023-11-24T06:08:02.405217+0000","last_peered":"2023-11-24T06:08:02.405217+0000","last_clean":"2023-11-24T06:08:02.405217+0000","last_became_active":"2023-11-24T06:08:00.422721+0000","last_became_peered":"2023-11-24T06:08:00.422721+0000","last_unstale":"2023-11-24T06:08:02.405217+0000","last_undegraded":"2023-11-24T06:08:02.405217+0000","last_fullsized":"2023-11-24T06:08:02.405217+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:29.195643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.427283+0000","last_change":"2023-11-24T06:08:01.424141+0000","last_active":"2023-11-24T06:08:02.427283+0000","last_peered":"2023-11-24T06:08:02.427283+0000","last_clean":"2023-11-24T06:08:02.427283+0000","last_became_active":"2023-11-24T06:08:01.423837+0000","last_became_peered":"2023-11-24T06:08:01.423837+0000","last_unstale":"2023-11-24T06:08:02.427283+0000","last_undegraded":"2023-11-24T06:08:02.427283+0000","last_fullsized":"2023-11-24T06:08:02.427283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:40:20.503614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.407768+0000","last_change":"2023-11-24T06:08:00.420556+0000","last_active":"2023-11-24T06:08:02.407768+0000","last_peered":"2023-11-24T06:08:02.407768+0000","last_clean":"2023-11-24T06:08:02.407768+0000","last_became_active":"2023-11-24T06:08:00.420250+0000","last_became_peered":"2023-11-24T06:08:00.420250+0000","last_unstale":"2023-11-24T06:08:02.407768+0000","last_undegraded":"2023-11-24T06:08:02.407768+0000","last_fullsized":"2023-11-24T06:08:02.407768+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:11.429196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859838+0000","last_change":"2023-11-24T06:08:01.440217+0000","last_active":"2023-11-24T06:08:02.859838+0000","last_peered":"2023-11-24T06:08:02.859838+0000","last_clean":"2023-11-24T06:08:02.859838+0000","last_became_active":"2023-11-24T06:08:01.439942+0000","last_became_peered":"2023-11-24T06:08:01.439942+0000","last_unstale":"2023-11-24T06:08:02.859838+0000","last_undegraded":"2023-11-24T06:08:02.859838+0000","last_fullsized":"2023-11-24T06:08:02.859838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:26:20.272046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.404803+0000","last_change":"2023-11-24T06:08:00.413735+0000","last_active":"2023-11-24T06:08:02.404803+0000","last_peered":"2023-11-24T06:08:02.404803+0000","last_clean":"2023-11-24T06:08:02.404803+0000","last_became_active":"2023-11-24T06:08:00.413375+0000","last_became_peered":"2023-11-24T06:08:00.413375+0000","last_unstale":"2023-11-24T06:08:02.404803+0000","last_undegraded":"2023-11-24T06:08:02.404803+0000","last_fullsized":"2023-11-24T06:08:02.404803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:20.079860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858677+0000","last_change":"2023-11-24T06:08:01.435772+0000","last_active":"2023-11-24T06:08:02.858677+0000","last_peered":"2023-11-24T06:08:02.858677+0000","last_clean":"2023-11-24T06:08:02.858677+0000","last_became_active":"2023-11-24T06:08:01.435495+0000","last_became_peered":"2023-11-24T06:08:01.435495+0000","last_unstale":"2023-11-24T06:08:02.858677+0000","last_undegraded":"2023-11-24T06:08:02.858677+0000","last_fullsized":"2023-11-24T06:08:02.858677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:01.832945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.407811+0000","last_change":"2023-11-24T06:08:00.415129+0000","last_active":"2023-11-24T06:08:02.407811+0000","last_peered":"2023-11-24T06:08:02.407811+0000","last_clean":"2023-11-24T06:08:02.407811+0000","last_became_active":"2023-11-24T06:08:00.414888+0000","last_became_peered":"2023-11-24T06:08:00.414888+0000","last_unstale":"2023-11-24T06:08:02.407811+0000","last_undegraded":"2023-11-24T06:08:02.407811+0000","last_fullsized":"2023-11-24T06:08:02.407811+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:41.551384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859426+0000","last_change":"2023-11-24T06:08:01.442379+0000","last_active":"2023-11-24T06:08:02.859426+0000","last_peered":"2023-11-24T06:08:02.859426+0000","last_clean":"2023-11-24T06:08:02.859426+0000","last_became_active":"2023-11-24T06:08:01.442109+0000","last_became_peered":"2023-11-24T06:08:01.442109+0000","last_unstale":"2023-11-24T06:08:02.859426+0000","last_undegraded":"2023-11-24T06:08:02.859426+0000","last_fullsized":"2023-11-24T06:08:02.859426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:19:13.981084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.427230+0000","last_change":"2023-11-24T06:08:01.429866+0000","last_active":"2023-11-24T06:08:02.427230+0000","last_peered":"2023-11-24T06:08:02.427230+0000","last_clean":"2023-11-24T06:08:02.427230+0000","last_became_active":"2023-11-24T06:08:01.429446+0000","last_became_peered":"2023-11-24T06:08:01.429446+0000","last_unstale":"2023-11-24T06:08:02.427230+0000","last_undegraded":"2023-11-24T06:08:02.427230+0000","last_fullsized":"2023-11-24T06:08:02.427230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:52:57.033291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.429315+0000","last_change":"2023-11-24T06:08:01.435709+0000","last_active":"2023-11-24T06:08:02.429315+0000","last_peered":"2023-11-24T06:08:02.429315+0000","last_clean":"2023-11-24T06:08:02.429315+0000","last_became_active":"2023-11-24T06:08:01.435488+0000","last_became_peered":"2023-11-24T06:08:01.435488+0000","last_unstale":"2023-11-24T06:08:02.429315+0000","last_undegraded":"2023-11-24T06:08:02.429315+0000","last_fullsized":"2023-11-24T06:08:02.429315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:16.537475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859231+0000","last_change":"2023-11-24T06:08:01.434372+0000","last_active":"2023-11-24T06:08:02.859231+0000","last_peered":"2023-11-24T06:08:02.859231+0000","last_clean":"2023-11-24T06:08:02.859231+0000","last_became_active":"2023-11-24T06:08:01.433776+0000","last_became_peered":"2023-11-24T06:08:01.433776+0000","last_unstale":"2023-11-24T06:08:02.859231+0000","last_undegraded":"2023-11-24T06:08:02.859231+0000","last_fullsized":"2023-11-24T06:08:02.859231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:55:45.332918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858468+0000","last_change":"2023-11-24T06:08:01.433754+0000","last_active":"2023-11-24T06:08:02.858468+0000","last_peered":"2023-11-24T06:08:02.858468+0000","last_clean":"2023-11-24T06:08:02.858468+0000","last_became_active":"2023-11-24T06:08:01.433473+0000","last_became_peered":"2023-11-24T06:08:01.433473+0000","last_unstale":"2023-11-24T06:08:02.858468+0000","last_undegraded":"2023-11-24T06:08:02.858468+0000","last_fullsized":"2023-11-24T06:08:02.858468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:47:16.508162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858224+0000","last_change":"2023-11-24T06:08:01.428520+0000","last_active":"2023-11-24T06:08:02.858224+0000","last_peered":"2023-11-24T06:08:02.858224+0000","last_clean":"2023-11-24T06:08:02.858224+0000","last_became_active":"2023-11-24T06:08:01.428236+0000","last_became_peered":"2023-11-24T06:08:01.428236+0000","last_unstale":"2023-11-24T06:08:02.858224+0000","last_undegraded":"2023-11-24T06:08:02.858224+0000","last_fullsized":"2023-11-24T06:08:02.858224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:01.832945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.428655+0000","last_change":"2023-11-24T06:08:01.433972+0000","last_active":"2023-11-24T06:08:02.428655+0000","last_peered":"2023-11-24T06:08:02.428655+0000","last_clean":"2023-11-24T06:08:02.428655+0000","last_became_active":"2023-11-24T06:08:01.433685+0000","last_became_peered":"2023-11-24T06:08:01.433685+0000","last_unstale":"2023-11-24T06:08:02.428655+0000","last_undegraded":"2023-11-24T06:08:02.428655+0000","last_fullsized":"2023-11-24T06:08:02.428655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:52.422428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.427720+0000","last_change":"2023-11-24T06:08:01.435802+0000","last_active":"2023-11-24T06:08:02.427720+0000","last_peered":"2023-11-24T06:08:02.427720+0000","last_clean":"2023-11-24T06:08:02.427720+0000","last_became_active":"2023-11-24T06:08:01.435228+0000","last_became_peered":"2023-11-24T06:08:01.435228+0000","last_unstale":"2023-11-24T06:08:02.427720+0000","last_undegraded":"2023-11-24T06:08:02.427720+0000","last_fullsized":"2023-11-24T06:08:02.427720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:52.422428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858155+0000","last_change":"2023-11-24T06:08:01.432295+0000","last_active":"2023-11-24T06:08:02.858155+0000","last_peered":"2023-11-24T06:08:02.858155+0000","last_clean":"2023-11-24T06:08:02.858155+0000","last_became_active":"2023-11-24T06:08:01.431767+0000","last_became_peered":"2023-11-24T06:08:01.431767+0000","last_unstale":"2023-11-24T06:08:02.858155+0000","last_undegraded":"2023-11-24T06:08:02.858155+0000","last_fullsized":"2023-11-24T06:08:02.858155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:52.422428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858440+0000","last_change":"2023-11-24T06:08:01.434132+0000","last_active":"2023-11-24T06:08:02.858440+0000","last_peered":"2023-11-24T06:08:02.858440+0000","last_clean":"2023-11-24T06:08:02.858440+0000","last_became_active":"2023-11-24T06:08:01.433678+0000","last_became_peered":"2023-11-24T06:08:01.433678+0000","last_unstale":"2023-11-24T06:08:02.858440+0000","last_undegraded":"2023-11-24T06:08:02.858440+0000","last_fullsized":"2023-11-24T06:08:02.858440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:52.422428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.857901+0000","last_change":"2023-11-24T06:08:01.424409+0000","last_active":"2023-11-24T06:08:02.857901+0000","last_peered":"2023-11-24T06:08:02.857901+0000","last_clean":"2023-11-24T06:08:02.857901+0000","last_became_active":"2023-11-24T06:08:01.424152+0000","last_became_peered":"2023-11-24T06:08:01.424152+0000","last_unstale":"2023-11-24T06:08:02.857901+0000","last_undegraded":"2023-11-24T06:08:02.857901+0000","last_fullsized":"2023-11-24T06:08:02.857901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:40:20.503614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859100+0000","last_change":"2023-11-24T06:08:01.436895+0000","last_active":"2023-11-24T06:08:02.859100+0000","last_peered":"2023-11-24T06:08:02.859100+0000","last_clean":"2023-11-24T06:08:02.859100+0000","last_became_active":"2023-11-24T06:08:01.436631+0000","last_became_peered":"2023-11-24T06:08:01.436631+0000","last_unstale":"2023-11-24T06:08:02.859100+0000","last_undegraded":"2023-11-24T06:08:02.859100+0000","last_fullsized":"2023-11-24T06:08:02.859100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:32:41.812446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858334+0000","last_change":"2023-11-24T06:08:01.435442+0000","last_active":"2023-11-24T06:08:02.858334+0000","last_peered":"2023-11-24T06:08:02.858334+0000","last_clean":"2023-11-24T06:08:02.858334+0000","last_became_active":"2023-11-24T06:08:01.435135+0000","last_became_peered":"2023-11-24T06:08:01.435135+0000","last_unstale":"2023-11-24T06:08:02.858334+0000","last_undegraded":"2023-11-24T06:08:02.858334+0000","last_fullsized":"2023-11-24T06:08:02.858334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:59.812992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.427573+0000","last_change":"2023-11-24T06:08:01.423313+0000","last_active":"2023-11-24T06:08:02.427573+0000","last_peered":"2023-11-24T06:08:02.427573+0000","last_clean":"2023-11-24T06:08:02.427573+0000","last_became_active":"2023-11-24T06:08:01.423042+0000","last_became_peered":"2023-11-24T06:08:01.423042+0000","last_unstale":"2023-11-24T06:08:02.427573+0000","last_undegraded":"2023-11-24T06:08:02.427573+0000","last_fullsized":"2023-11-24T06:08:02.427573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:47:16.508162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.7670000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9529999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9640000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.048}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":661916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:08:10.480 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:08:10.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.633+0000 7f405571d700 1 Processor -- start 2023-11-24T06:08:10.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.633+0000 7f405571d700 1 -- start start 2023-11-24T06:08:10.640 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f405571d700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f405015e440 0x7f40500ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:10.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f405571d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40500d2750 0x7f40500ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:10.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f405571d700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40500cd150 con 0x7f405015e060 2023-11-24T06:08:10.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f405571d700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40500cfe80 con 0x7f405015e440 2023-11-24T06:08:10.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f405571d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40500ccb00 con 0x7f40500d2750 2023-11-24T06:08:10.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404effd700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f405015e060 0x7f40500cd5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53132/0 (socket says 172.21.15.91:53132) 2023-11-24T06:08:10.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404effd700 1 -- 172.21.15.91:0/2605088731 learned_addr learned my addr 172.21.15.91:0/2605088731 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:10.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404e7fc700 1 --2- 172.21.15.91:0/2605088731 >> v2:172.21.15.94:3300/0 conn(0x7f405015e440 0x7f40500ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:10.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404f7fe700 1 --2- 172.21.15.91:0/2605088731 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40500d2750 0x7f40500ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:10.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404dffb700 1 -- 172.21.15.91:0/2605088731 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667943248 0 0) 0x7f40500cd150 con 0x7f405015e060 2023-11-24T06:08:10.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404dffb700 1 -- 172.21.15.91:0/2605088731 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40380035b0 con 0x7f405015e060 2023-11-24T06:08:10.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404dffb700 1 -- 172.21.15.91:0/2605088731 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1563455849 0 0) 0x7f40380035b0 con 0x7f405015e060 2023-11-24T06:08:10.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404dffb700 1 -- 172.21.15.91:0/2605088731 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40500d2750 msgr2=0x7f40500ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:10.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404dffb700 1 --2- 172.21.15.91:0/2605088731 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40500d2750 0x7f40500ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404dffb700 1 -- 172.21.15.91:0/2605088731 >> v2:172.21.15.94:3300/0 conn(0x7f405015e440 msgr2=0x7f40500ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:10.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404dffb700 1 --2- 172.21.15.91:0/2605088731 >> v2:172.21.15.94:3300/0 conn(0x7f405015e440 0x7f40500ce5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404dffb700 1 -- 172.21.15.91:0/2605088731 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40500ccde0 con 0x7f405015e060 2023-11-24T06:08:10.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404dffb700 1 -- 172.21.15.91:0/2605088731 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f4040001670 con 0x7f405015e060 2023-11-24T06:08:10.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404f7fe700 1 --2- 172.21.15.91:0/2605088731 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40500d2750 0x7f40500ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:08:10.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404dffb700 1 -- 172.21.15.91:0/2605088731 >> v1:172.21.15.91:6789/0 conn(0x7f405015e060 legacy=0x7f40500cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404dffb700 1 --2- 172.21.15.91:0/2605088731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4038003a00 0x7f4038003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:10.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404dffb700 1 --2- 172.21.15.91:0/2605088731 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40500d2750 0x7f4038004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:10.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404dffb700 1 --2- 172.21.15.91:0/2605088731 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4038004d90 0x7f4038005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:10.642 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404dffb700 1 -- 172.21.15.91:0/2605088731 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40500cfe80 con 0x7f4038004d90 2023-11-24T06:08:10.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404dffb700 1 -- 172.21.15.91:0/2605088731 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40500ccb00 con 0x7f4038003a00 2023-11-24T06:08:10.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404dffb700 1 -- 172.21.15.91:0/2605088731 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403800ba10 con 0x7f40500d2750 2023-11-24T06:08:10.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404e7fc700 1 --2- 172.21.15.91:0/2605088731 >> v2:172.21.15.94:3300/0 conn(0x7f405015e440 0x7f40500ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:08:10.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404f7fe700 1 --2- 172.21.15.91:0/2605088731 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4038004d90 0x7f4038005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:10.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.634+0000 7f404effd700 1 --2- 172.21.15.91:0/2605088731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4038003a00 0x7f4038003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:10.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.635+0000 7f404e7fc700 1 --2- 172.21.15.91:0/2605088731 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40500d2750 0x7f4038004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:10.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.635+0000 7f404f7fe700 1 -- 172.21.15.91:0/2605088731 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40500d2750 msgr2=0x7f4038004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:10.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.635+0000 7f404f7fe700 1 --2- 172.21.15.91:0/2605088731 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40500d2750 0x7f4038004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.635+0000 7f404f7fe700 1 -- 172.21.15.91:0/2605088731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4038003a00 msgr2=0x7f4038003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:10.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.635+0000 7f404f7fe700 1 --2- 172.21.15.91:0/2605088731 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4038003a00 0x7f4038003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.635+0000 7f404f7fe700 1 -- 172.21.15.91:0/2605088731 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4048009a70 con 0x7f4038004d90 2023-11-24T06:08:10.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.635+0000 7f404e7fc700 1 --2- 172.21.15.91:0/2605088731 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40500d2750 0x7f4038004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:10.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.635+0000 7f404f7fe700 1 --2- 172.21.15.91:0/2605088731 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4038004d90 0x7f4038005210 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f404800e350 tx=0x7f4038005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5de5e735c9f2fa1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:10.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.635+0000 7f404dffb700 1 -- 172.21.15.91:0/2605088731 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404800ed90 con 0x7f4038004d90 2023-11-24T06:08:10.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.635+0000 7f404dffb700 1 -- 172.21.15.91:0/2605088731 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404800f040 con 0x7f4038004d90 2023-11-24T06:08:10.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.635+0000 7f404dffb700 1 -- 172.21.15.91:0/2605088731 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40480146a0 con 0x7f4038004d90 2023-11-24T06:08:10.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.635+0000 7f405571d700 1 -- 172.21.15.91:0/2605088731 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4038004d90 msgr2=0x7f4038005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.635+0000 7f405571d700 1 --2- 172.21.15.91:0/2605088731 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4038004d90 0x7f4038005210 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f404800e350 tx=0x7f4038005880 comp rx=0 tx=0).stop 2023-11-24T06:08:10.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.635+0000 7f404f7fe700 1 -- 172.21.15.91:0/2605088731 reap_dead start 2023-11-24T06:08:10.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.635+0000 7f405571d700 1 -- 172.21.15.91:0/2605088731 shutdown_connections 2023-11-24T06:08:10.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.635+0000 7f405571d700 1 -- 172.21.15.91:0/2605088731 >> 172.21.15.91:0/2605088731 conn(0x7f40500d1c30 msgr2=0x7f40500d2040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:10.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.636+0000 7f405571d700 1 -- 172.21.15.91:0/2605088731 shutdown_connections 2023-11-24T06:08:10.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.636+0000 7f405571d700 1 -- 172.21.15.91:0/2605088731 wait complete. 2023-11-24T06:08:10.645 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.637+0000 7f405571d700 1 Processor -- start 2023-11-24T06:08:10.645 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.637+0000 7f405571d700 1 -- start start 2023-11-24T06:08:10.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f405571d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40501f33f0 0x7f40500b65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:10.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f405571d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40500b4a80 0x7f40500b4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:10.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f405571d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f40500ce340 0x7f40500ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:10.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f405571d700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40500ced00 con 0x7f40500ce340 2023-11-24T06:08:10.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f405571d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40500b6110 con 0x7f40501f33f0 2023-11-24T06:08:10.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f405571d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40500b5500 con 0x7f40500b4a80 2023-11-24T06:08:10.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f404f7fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f40500ce340 0x7f40500ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:10.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f404f7fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f40500ce340 0x7f40500ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58214/0 (socket says 172.21.15.91:58214) 2023-11-24T06:08:10.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f404f7fe700 1 -- 172.21.15.91:0/3531897618 learned_addr learned my addr 172.21.15.91:0/3531897618 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:10.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f404e7fc700 1 --2- 172.21.15.91:0/3531897618 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40500b4a80 0x7f40500b4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:10.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f404effd700 1 --2- 172.21.15.91:0/3531897618 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40501f33f0 0x7f40500b65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:10.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f404f7fe700 1 -- 172.21.15.91:0/3531897618 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40500b4a80 msgr2=0x7f40500b4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f404f7fe700 1 --2- 172.21.15.91:0/3531897618 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40500b4a80 0x7f40500b4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f404f7fe700 1 -- 172.21.15.91:0/3531897618 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40501f33f0 msgr2=0x7f40500b65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f404f7fe700 1 --2- 172.21.15.91:0/3531897618 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40501f33f0 0x7f40500b65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f404f7fe700 1 -- 172.21.15.91:0/3531897618 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40500b57e0 con 0x7f40500ce340 2023-11-24T06:08:10.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f404effd700 1 --2- 172.21.15.91:0/3531897618 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40501f33f0 0x7f40500b65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:10.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f404f7fe700 1 --2- 172.21.15.91:0/3531897618 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f40500ce340 0x7f40500ce7c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f40380046f0 tx=0x7f4048009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:10.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f40377fe700 1 -- 172.21.15.91:0/3531897618 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4048007000 con 0x7f40500ce340 2023-11-24T06:08:10.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f40377fe700 1 -- 172.21.15.91:0/3531897618 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4048004430 con 0x7f40500ce340 2023-11-24T06:08:10.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f405571d700 1 -- 172.21.15.91:0/3531897618 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40500b5ac0 con 0x7f40500ce340 2023-11-24T06:08:10.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f40377fe700 1 -- 172.21.15.91:0/3531897618 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40480047e0 con 0x7f40500ce340 2023-11-24T06:08:10.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.638+0000 7f405571d700 1 -- 172.21.15.91:0/3531897618 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f405014fe90 con 0x7f40500ce340 2023-11-24T06:08:10.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.639+0000 7f405571d700 1 -- 172.21.15.91:0/3531897618 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4050067f60 con 0x7f40500ce340 2023-11-24T06:08:10.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.640+0000 7f40377fe700 1 -- 172.21.15.91:0/3531897618 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4048024070 con 0x7f40500ce340 2023-11-24T06:08:10.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.640+0000 7f40377fe700 1 --2- 172.21.15.91:0/3531897618 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f403c06aed0 0x7f403c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:10.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.640+0000 7f40377fe700 1 -- 172.21.15.91:0/3531897618 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f404808b4f0 con 0x7f40500ce340 2023-11-24T06:08:10.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.640+0000 7f404effd700 1 --2- 172.21.15.91:0/3531897618 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f403c06aed0 0x7f403c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:10.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.640+0000 7f404effd700 1 --2- 172.21.15.91:0/3531897618 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f403c06aed0 0x7f403c06d390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4040001050 tx=0x7f4040009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:10.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.644+0000 7f40377fe700 1 -- 172.21.15.91:0/3531897618 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4048056630 con 0x7f40500ce340 2023-11-24T06:08:10.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.835+0000 7f405571d700 1 -- 172.21.15.91:0/3531897618 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f40500cfa10 con 0x7f403c06aed0 2023-11-24T06:08:10.846 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.842+0000 7f40377fe700 1 -- 172.21.15.91:0/3531897618 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264531 (secure 0 0 0) 0x7f40400070e0 con 0x7f403c06aed0 2023-11-24T06:08:10.849 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:10.851 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.845+0000 7f405571d700 1 -- 172.21.15.91:0/3531897618 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f403c06aed0 msgr2=0x7f403c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.851 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.845+0000 7f405571d700 1 --2- 172.21.15.91:0/3531897618 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f403c06aed0 0x7f403c06d390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4040001050 tx=0x7f4040009040 comp rx=0 tx=0).stop 2023-11-24T06:08:10.851 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.845+0000 7f405571d700 1 -- 172.21.15.91:0/3531897618 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f40500ce340 msgr2=0x7f40500ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:10.851 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.845+0000 7f405571d700 1 --2- 172.21.15.91:0/3531897618 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f40500ce340 0x7f40500ce7c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f40380046f0 tx=0x7f4048009750 comp rx=0 tx=0).stop 2023-11-24T06:08:10.851 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.845+0000 7f405571d700 1 -- 172.21.15.91:0/3531897618 shutdown_connections 2023-11-24T06:08:10.851 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.845+0000 7f405571d700 1 --2- 172.21.15.91:0/3531897618 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f403c06aed0 0x7f403c06d390 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.851 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.845+0000 7f405571d700 1 --2- 172.21.15.91:0/3531897618 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40501f33f0 0x7f40500b65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.845+0000 7f405571d700 1 --2- 172.21.15.91:0/3531897618 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f40500ce340 0x7f40500ce7c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.845+0000 7f405571d700 1 --2- 172.21.15.91:0/3531897618 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40500b4a80 0x7f40500b4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:10.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.845+0000 7f405571d700 1 -- 172.21.15.91:0/3531897618 >> 172.21.15.91:0/3531897618 conn(0x7f40500d1c30 msgr2=0x7f40500cd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:10.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.845+0000 7f405571d700 1 -- 172.21.15.91:0/3531897618 shutdown_connections 2023-11-24T06:08:10.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:10.845+0000 7f405571d700 1 -- 172.21.15.91:0/3531897618 wait complete. 2023-11-24T06:08:10.852 INFO:teuthology.orchestra.run.smithi091.stderr:dumped all 2023-11-24T06:08:10.861 INFO:teuthology.orchestra.run.smithi091.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-11-24T06:08:09.398632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219012,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755708,"statfs":{"total":773094113280,"available":772869844992,"internally_reserved":0,"allocated":2891776,"data_stored":1752836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005345"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859587+0000","last_change":"2023-11-24T06:08:01.435066+0000","last_active":"2023-11-24T06:08:02.859587+0000","last_peered":"2023-11-24T06:08:02.859587+0000","last_clean":"2023-11-24T06:08:02.859587+0000","last_became_active":"2023-11-24T06:08:01.434733+0000","last_became_peered":"2023-11-24T06:08:01.434733+0000","last_unstale":"2023-11-24T06:08:02.859587+0000","last_undegraded":"2023-11-24T06:08:02.859587+0000","last_fullsized":"2023-11-24T06:08:02.859587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:46:08.238325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.428542+0000","last_change":"2023-11-24T06:08:01.428588+0000","last_active":"2023-11-24T06:08:02.428542+0000","last_peered":"2023-11-24T06:08:02.428542+0000","last_clean":"2023-11-24T06:08:02.428542+0000","last_became_active":"2023-11-24T06:08:01.428257+0000","last_became_peered":"2023-11-24T06:08:01.428257+0000","last_unstale":"2023-11-24T06:08:02.428542+0000","last_undegraded":"2023-11-24T06:08:02.428542+0000","last_fullsized":"2023-11-24T06:08:02.428542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:40:20.503614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858341+0000","last_change":"2023-11-24T06:08:01.428960+0000","last_active":"2023-11-24T06:08:02.858341+0000","last_peered":"2023-11-24T06:08:02.858341+0000","last_clean":"2023-11-24T06:08:02.858341+0000","last_became_active":"2023-11-24T06:08:01.428600+0000","last_became_peered":"2023-11-24T06:08:01.428600+0000","last_unstale":"2023-11-24T06:08:02.858341+0000","last_undegraded":"2023-11-24T06:08:02.858341+0000","last_fullsized":"2023-11-24T06:08:02.858341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:52:57.033291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858818+0000","last_change":"2023-11-24T06:08:01.435529+0000","last_active":"2023-11-24T06:08:02.858818+0000","last_peered":"2023-11-24T06:08:02.858818+0000","last_clean":"2023-11-24T06:08:02.858818+0000","last_became_active":"2023-11-24T06:08:01.435128+0000","last_became_peered":"2023-11-24T06:08:01.435128+0000","last_unstale":"2023-11-24T06:08:02.858818+0000","last_undegraded":"2023-11-24T06:08:02.858818+0000","last_fullsized":"2023-11-24T06:08:02.858818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:06:58.673664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859171+0000","last_change":"2023-11-24T06:08:01.434208+0000","last_active":"2023-11-24T06:08:02.859171+0000","last_peered":"2023-11-24T06:08:02.859171+0000","last_clean":"2023-11-24T06:08:02.859171+0000","last_became_active":"2023-11-24T06:08:01.433849+0000","last_became_peered":"2023-11-24T06:08:01.433849+0000","last_unstale":"2023-11-24T06:08:02.859171+0000","last_undegraded":"2023-11-24T06:08:02.859171+0000","last_fullsized":"2023-11-24T06:08:02.859171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:38:12.787229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858655+0000","last_change":"2023-11-24T06:08:01.436361+0000","last_active":"2023-11-24T06:08:02.858655+0000","last_peered":"2023-11-24T06:08:02.858655+0000","last_clean":"2023-11-24T06:08:02.858655+0000","last_became_active":"2023-11-24T06:08:01.434166+0000","last_became_peered":"2023-11-24T06:08:01.434166+0000","last_unstale":"2023-11-24T06:08:02.858655+0000","last_undegraded":"2023-11-24T06:08:02.858655+0000","last_fullsized":"2023-11-24T06:08:02.858655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:22.215190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858814+0000","last_change":"2023-11-24T06:08:01.433541+0000","last_active":"2023-11-24T06:08:02.858814+0000","last_peered":"2023-11-24T06:08:02.858814+0000","last_clean":"2023-11-24T06:08:02.858814+0000","last_became_active":"2023-11-24T06:08:01.433237+0000","last_became_peered":"2023-11-24T06:08:01.433237+0000","last_unstale":"2023-11-24T06:08:02.858814+0000","last_undegraded":"2023-11-24T06:08:02.858814+0000","last_fullsized":"2023-11-24T06:08:02.858814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:32:41.812446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.857978+0000","last_change":"2023-11-24T06:08:01.435678+0000","last_active":"2023-11-24T06:08:02.857978+0000","last_peered":"2023-11-24T06:08:02.857978+0000","last_clean":"2023-11-24T06:08:02.857978+0000","last_became_active":"2023-11-24T06:08:01.435365+0000","last_became_peered":"2023-11-24T06:08:01.435365+0000","last_unstale":"2023-11-24T06:08:02.857978+0000","last_undegraded":"2023-11-24T06:08:02.857978+0000","last_fullsized":"2023-11-24T06:08:02.857978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:55:45.332918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859541+0000","last_change":"2023-11-24T06:08:01.438757+0000","last_active":"2023-11-24T06:08:02.859541+0000","last_peered":"2023-11-24T06:08:02.859541+0000","last_clean":"2023-11-24T06:08:02.859541+0000","last_became_active":"2023-11-24T06:08:01.438495+0000","last_became_peered":"2023-11-24T06:08:01.438495+0000","last_unstale":"2023-11-24T06:08:02.859541+0000","last_undegraded":"2023-11-24T06:08:02.859541+0000","last_fullsized":"2023-11-24T06:08:02.859541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:51:58.808088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.427783+0000","last_change":"2023-11-24T06:08:01.435879+0000","last_active":"2023-11-24T06:08:02.427783+0000","last_peered":"2023-11-24T06:08:02.427783+0000","last_clean":"2023-11-24T06:08:02.427783+0000","last_became_active":"2023-11-24T06:08:01.435556+0000","last_became_peered":"2023-11-24T06:08:01.435556+0000","last_unstale":"2023-11-24T06:08:02.427783+0000","last_undegraded":"2023-11-24T06:08:02.427783+0000","last_fullsized":"2023-11-24T06:08:02.427783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:22.215190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.427548+0000","last_change":"2023-11-24T06:08:01.436002+0000","last_active":"2023-11-24T06:08:02.427548+0000","last_peered":"2023-11-24T06:08:02.427548+0000","last_clean":"2023-11-24T06:08:02.427548+0000","last_became_active":"2023-11-24T06:08:01.435547+0000","last_became_peered":"2023-11-24T06:08:01.435547+0000","last_unstale":"2023-11-24T06:08:02.427548+0000","last_undegraded":"2023-11-24T06:08:02.427548+0000","last_fullsized":"2023-11-24T06:08:02.427548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:28.564398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859683+0000","last_change":"2023-11-24T06:08:01.431781+0000","last_active":"2023-11-24T06:08:02.859683+0000","last_peered":"2023-11-24T06:08:02.859683+0000","last_clean":"2023-11-24T06:08:02.859683+0000","last_became_active":"2023-11-24T06:08:01.431494+0000","last_became_peered":"2023-11-24T06:08:01.431494+0000","last_unstale":"2023-11-24T06:08:02.859683+0000","last_undegraded":"2023-11-24T06:08:02.859683+0000","last_fullsized":"2023-11-24T06:08:02.859683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:37:39.086654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.857833+0000","last_change":"2023-11-24T06:08:01.434455+0000","last_active":"2023-11-24T06:08:02.857833+0000","last_peered":"2023-11-24T06:08:02.857833+0000","last_clean":"2023-11-24T06:08:02.857833+0000","last_became_active":"2023-11-24T06:08:01.434201+0000","last_became_peered":"2023-11-24T06:08:01.434201+0000","last_unstale":"2023-11-24T06:08:02.857833+0000","last_undegraded":"2023-11-24T06:08:02.857833+0000","last_fullsized":"2023-11-24T06:08:02.857833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:52:57.033291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.857927+0000","last_change":"2023-11-24T06:08:01.427810+0000","last_active":"2023-11-24T06:08:02.857927+0000","last_peered":"2023-11-24T06:08:02.857927+0000","last_clean":"2023-11-24T06:08:02.857927+0000","last_became_active":"2023-11-24T06:08:01.427545+0000","last_became_peered":"2023-11-24T06:08:01.427545+0000","last_unstale":"2023-11-24T06:08:02.857927+0000","last_undegraded":"2023-11-24T06:08:02.857927+0000","last_fullsized":"2023-11-24T06:08:02.857927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:30.200581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.428420+0000","last_change":"2023-11-24T06:08:01.431162+0000","last_active":"2023-11-24T06:08:02.428420+0000","last_peered":"2023-11-24T06:08:02.428420+0000","last_clean":"2023-11-24T06:08:02.428420+0000","last_became_active":"2023-11-24T06:08:01.430862+0000","last_became_peered":"2023-11-24T06:08:01.430862+0000","last_unstale":"2023-11-24T06:08:02.428420+0000","last_undegraded":"2023-11-24T06:08:02.428420+0000","last_fullsized":"2023-11-24T06:08:02.428420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:52:57.033291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858252+0000","last_change":"2023-11-24T06:08:01.434078+0000","last_active":"2023-11-24T06:08:02.858252+0000","last_peered":"2023-11-24T06:08:02.858252+0000","last_clean":"2023-11-24T06:08:02.858252+0000","last_became_active":"2023-11-24T06:08:01.433636+0000","last_became_peered":"2023-11-24T06:08:01.433636+0000","last_unstale":"2023-11-24T06:08:02.858252+0000","last_undegraded":"2023-11-24T06:08:02.858252+0000","last_fullsized":"2023-11-24T06:08:02.858252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:47:16.508162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859546+0000","last_change":"2023-11-24T06:08:01.436588+0000","last_active":"2023-11-24T06:08:02.859546+0000","last_peered":"2023-11-24T06:08:02.859546+0000","last_clean":"2023-11-24T06:08:02.859546+0000","last_became_active":"2023-11-24T06:08:01.436193+0000","last_became_peered":"2023-11-24T06:08:01.436193+0000","last_unstale":"2023-11-24T06:08:02.859546+0000","last_undegraded":"2023-11-24T06:08:02.859546+0000","last_fullsized":"2023-11-24T06:08:02.859546+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:51:58.808088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.429338+0000","last_change":"2023-11-24T06:08:01.424485+0000","last_active":"2023-11-24T06:08:02.429338+0000","last_peered":"2023-11-24T06:08:02.429338+0000","last_clean":"2023-11-24T06:08:02.429338+0000","last_became_active":"2023-11-24T06:08:01.424234+0000","last_became_peered":"2023-11-24T06:08:01.424234+0000","last_unstale":"2023-11-24T06:08:02.429338+0000","last_undegraded":"2023-11-24T06:08:02.429338+0000","last_fullsized":"2023-11-24T06:08:02.429338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:29.649298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859691+0000","last_change":"2023-11-24T06:08:01.434319+0000","last_active":"2023-11-24T06:08:02.859691+0000","last_peered":"2023-11-24T06:08:02.859691+0000","last_clean":"2023-11-24T06:08:02.859691+0000","last_became_active":"2023-11-24T06:08:01.433984+0000","last_became_peered":"2023-11-24T06:08:01.433984+0000","last_unstale":"2023-11-24T06:08:02.859691+0000","last_undegraded":"2023-11-24T06:08:02.859691+0000","last_fullsized":"2023-11-24T06:08:02.859691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:30:33.401914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.407937+0000","last_change":"2023-11-24T06:08:00.418990+0000","last_active":"2023-11-24T06:08:02.407937+0000","last_peered":"2023-11-24T06:08:02.407937+0000","last_clean":"2023-11-24T06:08:02.407937+0000","last_became_active":"2023-11-24T06:08:00.418683+0000","last_became_peered":"2023-11-24T06:08:00.418683+0000","last_unstale":"2023-11-24T06:08:02.407937+0000","last_undegraded":"2023-11-24T06:08:02.407937+0000","last_fullsized":"2023-11-24T06:08:02.407937+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:13.152063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858033+0000","last_change":"2023-11-24T06:08:01.431913+0000","last_active":"2023-11-24T06:08:02.858033+0000","last_peered":"2023-11-24T06:08:02.858033+0000","last_clean":"2023-11-24T06:08:02.858033+0000","last_became_active":"2023-11-24T06:08:01.431373+0000","last_became_peered":"2023-11-24T06:08:01.431373+0000","last_unstale":"2023-11-24T06:08:02.858033+0000","last_undegraded":"2023-11-24T06:08:02.858033+0000","last_fullsized":"2023-11-24T06:08:02.858033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:28.564398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.406361+0000","last_change":"2023-11-24T06:08:00.424674+0000","last_active":"2023-11-24T06:08:02.406361+0000","last_peered":"2023-11-24T06:08:02.406361+0000","last_clean":"2023-11-24T06:08:02.406361+0000","last_became_active":"2023-11-24T06:08:00.424282+0000","last_became_peered":"2023-11-24T06:08:00.424282+0000","last_unstale":"2023-11-24T06:08:02.406361+0000","last_undegraded":"2023-11-24T06:08:02.406361+0000","last_fullsized":"2023-11-24T06:08:02.406361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:03.380514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858392+0000","last_change":"2023-11-24T06:08:01.434659+0000","last_active":"2023-11-24T06:08:02.858392+0000","last_peered":"2023-11-24T06:08:02.858392+0000","last_clean":"2023-11-24T06:08:02.858392+0000","last_became_active":"2023-11-24T06:08:01.434373+0000","last_became_peered":"2023-11-24T06:08:01.434373+0000","last_unstale":"2023-11-24T06:08:02.858392+0000","last_undegraded":"2023-11-24T06:08:02.858392+0000","last_fullsized":"2023-11-24T06:08:02.858392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:40:20.503614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.405297+0000","last_change":"2023-11-24T06:08:00.426586+0000","last_active":"2023-11-24T06:08:02.405297+0000","last_peered":"2023-11-24T06:08:02.405297+0000","last_clean":"2023-11-24T06:08:02.405297+0000","last_became_active":"2023-11-24T06:08:00.426279+0000","last_became_peered":"2023-11-24T06:08:00.426279+0000","last_unstale":"2023-11-24T06:08:02.405297+0000","last_undegraded":"2023-11-24T06:08:02.405297+0000","last_fullsized":"2023-11-24T06:08:02.405297+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:06:57.668726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858648+0000","last_change":"2023-11-24T06:08:01.439887+0000","last_active":"2023-11-24T06:08:02.858648+0000","last_peered":"2023-11-24T06:08:02.858648+0000","last_clean":"2023-11-24T06:08:02.858648+0000","last_became_active":"2023-11-24T06:08:01.439606+0000","last_became_peered":"2023-11-24T06:08:01.439606+0000","last_unstale":"2023-11-24T06:08:02.858648+0000","last_undegraded":"2023-11-24T06:08:02.858648+0000","last_fullsized":"2023-11-24T06:08:02.858648+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:16:08.901829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.405228+0000","last_change":"2023-11-24T06:08:00.423364+0000","last_active":"2023-11-24T06:08:02.405228+0000","last_peered":"2023-11-24T06:08:02.405228+0000","last_clean":"2023-11-24T06:08:02.405228+0000","last_became_active":"2023-11-24T06:08:00.423062+0000","last_became_peered":"2023-11-24T06:08:00.423062+0000","last_unstale":"2023-11-24T06:08:02.405228+0000","last_undegraded":"2023-11-24T06:08:02.405228+0000","last_fullsized":"2023-11-24T06:08:02.405228+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:28.644360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859170+0000","last_change":"2023-11-24T06:08:01.438369+0000","last_active":"2023-11-24T06:08:02.859170+0000","last_peered":"2023-11-24T06:08:02.859170+0000","last_clean":"2023-11-24T06:08:02.859170+0000","last_became_active":"2023-11-24T06:08:01.438112+0000","last_became_peered":"2023-11-24T06:08:01.438112+0000","last_unstale":"2023-11-24T06:08:02.859170+0000","last_undegraded":"2023-11-24T06:08:02.859170+0000","last_fullsized":"2023-11-24T06:08:02.859170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:02:53.563916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.406278+0000","last_change":"2023-11-24T06:08:00.425844+0000","last_active":"2023-11-24T06:08:02.406278+0000","last_peered":"2023-11-24T06:08:02.406278+0000","last_clean":"2023-11-24T06:08:02.406278+0000","last_became_active":"2023-11-24T06:08:00.425587+0000","last_became_peered":"2023-11-24T06:08:00.425587+0000","last_unstale":"2023-11-24T06:08:02.406278+0000","last_undegraded":"2023-11-24T06:08:02.406278+0000","last_fullsized":"2023-11-24T06:08:02.406278+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:29.195643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858441+0000","last_change":"2023-11-24T06:08:01.427389+0000","last_active":"2023-11-24T06:08:02.858441+0000","last_peered":"2023-11-24T06:08:02.858441+0000","last_clean":"2023-11-24T06:08:02.858441+0000","last_became_active":"2023-11-24T06:08:01.427038+0000","last_became_peered":"2023-11-24T06:08:01.427038+0000","last_unstale":"2023-11-24T06:08:02.858441+0000","last_undegraded":"2023-11-24T06:08:02.858441+0000","last_fullsized":"2023-11-24T06:08:02.858441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:26:32.299407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.402091+0000","last_change":"2023-11-24T06:08:00.411792+0000","last_active":"2023-11-24T06:08:02.402091+0000","last_peered":"2023-11-24T06:08:02.402091+0000","last_clean":"2023-11-24T06:08:02.402091+0000","last_became_active":"2023-11-24T06:08:00.411486+0000","last_became_peered":"2023-11-24T06:08:00.411486+0000","last_unstale":"2023-11-24T06:08:02.402091+0000","last_undegraded":"2023-11-24T06:08:02.402091+0000","last_fullsized":"2023-11-24T06:08:02.402091+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:41.551384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858551+0000","last_change":"2023-11-24T06:08:01.438231+0000","last_active":"2023-11-24T06:08:02.858551+0000","last_peered":"2023-11-24T06:08:02.858551+0000","last_clean":"2023-11-24T06:08:02.858551+0000","last_became_active":"2023-11-24T06:08:01.437944+0000","last_became_peered":"2023-11-24T06:08:01.437944+0000","last_unstale":"2023-11-24T06:08:02.858551+0000","last_undegraded":"2023-11-24T06:08:02.858551+0000","last_fullsized":"2023-11-24T06:08:02.858551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:22.215190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.406223+0000","last_change":"2023-11-24T06:08:00.416967+0000","last_active":"2023-11-24T06:08:02.406223+0000","last_peered":"2023-11-24T06:08:02.406223+0000","last_clean":"2023-11-24T06:08:02.406223+0000","last_became_active":"2023-11-24T06:08:00.416532+0000","last_became_peered":"2023-11-24T06:08:00.416532+0000","last_unstale":"2023-11-24T06:08:02.406223+0000","last_undegraded":"2023-11-24T06:08:02.406223+0000","last_fullsized":"2023-11-24T06:08:02.406223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:20.079860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858392+0000","last_change":"2023-11-24T06:08:01.429802+0000","last_active":"2023-11-24T06:08:02.858392+0000","last_peered":"2023-11-24T06:08:02.858392+0000","last_clean":"2023-11-24T06:08:02.858392+0000","last_became_active":"2023-11-24T06:08:01.429578+0000","last_became_peered":"2023-11-24T06:08:01.429578+0000","last_unstale":"2023-11-24T06:08:02.858392+0000","last_undegraded":"2023-11-24T06:08:02.858392+0000","last_fullsized":"2023-11-24T06:08:02.858392+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:32:41.812446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.401833+0000","last_change":"2023-11-24T06:08:00.422030+0000","last_active":"2023-11-24T06:08:02.401833+0000","last_peered":"2023-11-24T06:08:02.401833+0000","last_clean":"2023-11-24T06:08:02.401833+0000","last_became_active":"2023-11-24T06:08:00.421740+0000","last_became_peered":"2023-11-24T06:08:00.421740+0000","last_unstale":"2023-11-24T06:08:02.401833+0000","last_undegraded":"2023-11-24T06:08:02.401833+0000","last_fullsized":"2023-11-24T06:08:02.401833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:58.808054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.428563+0000","last_change":"2023-11-24T06:08:01.433496+0000","last_active":"2023-11-24T06:08:02.428563+0000","last_peered":"2023-11-24T06:08:02.428563+0000","last_clean":"2023-11-24T06:08:02.428563+0000","last_became_active":"2023-11-24T06:08:01.433238+0000","last_became_peered":"2023-11-24T06:08:01.433238+0000","last_unstale":"2023-11-24T06:08:02.428563+0000","last_undegraded":"2023-11-24T06:08:02.428563+0000","last_fullsized":"2023-11-24T06:08:02.428563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:28.564398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.402192+0000","last_change":"2023-11-24T06:08:00.422595+0000","last_active":"2023-11-24T06:08:02.402192+0000","last_peered":"2023-11-24T06:08:02.402192+0000","last_clean":"2023-11-24T06:08:02.402192+0000","last_became_active":"2023-11-24T06:08:00.422365+0000","last_became_peered":"2023-11-24T06:08:00.422365+0000","last_unstale":"2023-11-24T06:08:02.402192+0000","last_undegraded":"2023-11-24T06:08:02.402192+0000","last_fullsized":"2023-11-24T06:08:02.402192+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:39.360506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859469+0000","last_change":"2023-11-24T06:08:01.434890+0000","last_active":"2023-11-24T06:08:02.859469+0000","last_peered":"2023-11-24T06:08:02.859469+0000","last_clean":"2023-11-24T06:08:02.859469+0000","last_became_active":"2023-11-24T06:08:01.434706+0000","last_became_peered":"2023-11-24T06:08:01.434706+0000","last_unstale":"2023-11-24T06:08:02.859469+0000","last_undegraded":"2023-11-24T06:08:02.859469+0000","last_fullsized":"2023-11-24T06:08:02.859469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:19:13.981084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.402050+0000","last_change":"2023-11-24T06:08:00.423297+0000","last_active":"2023-11-24T06:08:02.402050+0000","last_peered":"2023-11-24T06:08:02.402050+0000","last_clean":"2023-11-24T06:08:02.402050+0000","last_became_active":"2023-11-24T06:08:00.423028+0000","last_became_peered":"2023-11-24T06:08:00.423028+0000","last_unstale":"2023-11-24T06:08:02.402050+0000","last_undegraded":"2023-11-24T06:08:02.402050+0000","last_fullsized":"2023-11-24T06:08:02.402050+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:37:38.081716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858516+0000","last_change":"2023-11-24T06:08:01.437725+0000","last_active":"2023-11-24T06:08:02.858516+0000","last_peered":"2023-11-24T06:08:02.858516+0000","last_clean":"2023-11-24T06:08:02.858516+0000","last_became_active":"2023-11-24T06:08:01.437464+0000","last_became_peered":"2023-11-24T06:08:01.437464+0000","last_unstale":"2023-11-24T06:08:02.858516+0000","last_undegraded":"2023-11-24T06:08:02.858516+0000","last_fullsized":"2023-11-24T06:08:02.858516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:52.422428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.404823+0000","last_change":"2023-11-24T06:08:00.421041+0000","last_active":"2023-11-24T06:08:02.404823+0000","last_peered":"2023-11-24T06:08:02.404823+0000","last_clean":"2023-11-24T06:08:02.404823+0000","last_became_active":"2023-11-24T06:08:00.420677+0000","last_became_peered":"2023-11-24T06:08:00.420677+0000","last_unstale":"2023-11-24T06:08:02.404823+0000","last_undegraded":"2023-11-24T06:08:02.404823+0000","last_fullsized":"2023-11-24T06:08:02.404823+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:29.195643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859100+0000","last_change":"2023-11-24T06:08:01.437971+0000","last_active":"2023-11-24T06:08:02.859100+0000","last_peered":"2023-11-24T06:08:02.859100+0000","last_clean":"2023-11-24T06:08:02.859100+0000","last_became_active":"2023-11-24T06:08:01.437640+0000","last_became_peered":"2023-11-24T06:08:01.437640+0000","last_unstale":"2023-11-24T06:08:02.859100+0000","last_undegraded":"2023-11-24T06:08:02.859100+0000","last_fullsized":"2023-11-24T06:08:02.859100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:16.537475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.402105+0000","last_change":"2023-11-24T06:08:00.418490+0000","last_active":"2023-11-24T06:08:02.402105+0000","last_peered":"2023-11-24T06:08:02.402105+0000","last_clean":"2023-11-24T06:08:02.402105+0000","last_became_active":"2023-11-24T06:08:00.418200+0000","last_became_peered":"2023-11-24T06:08:00.418200+0000","last_unstale":"2023-11-24T06:08:02.402105+0000","last_undegraded":"2023-11-24T06:08:02.402105+0000","last_fullsized":"2023-11-24T06:08:02.402105+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:03.380514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859707+0000","last_change":"2023-11-24T06:08:01.439461+0000","last_active":"2023-11-24T06:08:02.859707+0000","last_peered":"2023-11-24T06:08:02.859707+0000","last_clean":"2023-11-24T06:08:02.859707+0000","last_became_active":"2023-11-24T06:08:01.439210+0000","last_became_peered":"2023-11-24T06:08:01.439210+0000","last_unstale":"2023-11-24T06:08:02.859707+0000","last_undegraded":"2023-11-24T06:08:02.859707+0000","last_fullsized":"2023-11-24T06:08:02.859707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:49:31.031906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.407837+0000","last_change":"2023-11-24T06:08:00.423172+0000","last_active":"2023-11-24T06:08:02.407837+0000","last_peered":"2023-11-24T06:08:02.407837+0000","last_clean":"2023-11-24T06:08:02.407837+0000","last_became_active":"2023-11-24T06:08:00.422899+0000","last_became_peered":"2023-11-24T06:08:00.422899+0000","last_unstale":"2023-11-24T06:08:02.407837+0000","last_undegraded":"2023-11-24T06:08:02.407837+0000","last_fullsized":"2023-11-24T06:08:02.407837+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:17.143751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.857787+0000","last_change":"2023-11-24T06:08:01.435150+0000","last_active":"2023-11-24T06:08:02.857787+0000","last_peered":"2023-11-24T06:08:02.857787+0000","last_clean":"2023-11-24T06:08:02.857787+0000","last_became_active":"2023-11-24T06:08:01.434861+0000","last_became_peered":"2023-11-24T06:08:01.434861+0000","last_unstale":"2023-11-24T06:08:02.857787+0000","last_undegraded":"2023-11-24T06:08:02.857787+0000","last_fullsized":"2023-11-24T06:08:02.857787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:40.365444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.401913+0000","last_change":"2023-11-24T06:08:00.422660+0000","last_active":"2023-11-24T06:08:02.401913+0000","last_peered":"2023-11-24T06:08:02.401913+0000","last_clean":"2023-11-24T06:08:02.401913+0000","last_became_active":"2023-11-24T06:08:00.421922+0000","last_became_peered":"2023-11-24T06:08:00.421922+0000","last_unstale":"2023-11-24T06:08:02.401913+0000","last_undegraded":"2023-11-24T06:08:02.401913+0000","last_fullsized":"2023-11-24T06:08:02.401913+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:27.559460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859321+0000","last_change":"2023-11-24T06:08:01.442581+0000","last_active":"2023-11-24T06:08:02.859321+0000","last_peered":"2023-11-24T06:08:02.859321+0000","last_clean":"2023-11-24T06:08:02.859321+0000","last_became_active":"2023-11-24T06:08:01.442145+0000","last_became_peered":"2023-11-24T06:08:01.442145+0000","last_unstale":"2023-11-24T06:08:02.859321+0000","last_undegraded":"2023-11-24T06:08:02.859321+0000","last_fullsized":"2023-11-24T06:08:02.859321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:16:08.901829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.402064+0000","last_change":"2023-11-24T06:08:00.413903+0000","last_active":"2023-11-24T06:08:02.402064+0000","last_peered":"2023-11-24T06:08:02.402064+0000","last_clean":"2023-11-24T06:08:02.402064+0000","last_became_active":"2023-11-24T06:08:00.413581+0000","last_became_peered":"2023-11-24T06:08:00.413581+0000","last_unstale":"2023-11-24T06:08:02.402064+0000","last_undegraded":"2023-11-24T06:08:02.402064+0000","last_fullsized":"2023-11-24T06:08:02.402064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:15.532537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.427222+0000","last_change":"2023-11-24T06:08:01.429649+0000","last_active":"2023-11-24T06:08:02.427222+0000","last_peered":"2023-11-24T06:08:02.427222+0000","last_clean":"2023-11-24T06:08:02.427222+0000","last_became_active":"2023-11-24T06:08:01.429189+0000","last_became_peered":"2023-11-24T06:08:01.429189+0000","last_unstale":"2023-11-24T06:08:02.427222+0000","last_undegraded":"2023-11-24T06:08:02.427222+0000","last_fullsized":"2023-11-24T06:08:02.427222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:59.812992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.407695+0000","last_change":"2023-11-24T06:08:00.417220+0000","last_active":"2023-11-24T06:08:02.407695+0000","last_peered":"2023-11-24T06:08:02.407695+0000","last_clean":"2023-11-24T06:08:02.407695+0000","last_became_active":"2023-11-24T06:08:00.416947+0000","last_became_peered":"2023-11-24T06:08:00.416947+0000","last_unstale":"2023-11-24T06:08:02.407695+0000","last_undegraded":"2023-11-24T06:08:02.407695+0000","last_fullsized":"2023-11-24T06:08:02.407695+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:19:18.816306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859012+0000","last_change":"2023-11-24T06:08:01.442182+0000","last_active":"2023-11-24T06:08:02.859012+0000","last_peered":"2023-11-24T06:08:02.859012+0000","last_clean":"2023-11-24T06:08:02.859012+0000","last_became_active":"2023-11-24T06:08:01.441895+0000","last_became_peered":"2023-11-24T06:08:01.441895+0000","last_unstale":"2023-11-24T06:08:02.859012+0000","last_undegraded":"2023-11-24T06:08:02.859012+0000","last_fullsized":"2023-11-24T06:08:02.859012+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:55:45.332918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.405064+0000","last_change":"2023-11-24T06:08:00.420431+0000","last_active":"2023-11-24T06:08:02.405064+0000","last_peered":"2023-11-24T06:08:02.405064+0000","last_clean":"2023-11-24T06:08:02.405064+0000","last_became_active":"2023-11-24T06:08:00.420108+0000","last_became_peered":"2023-11-24T06:08:00.420108+0000","last_unstale":"2023-11-24T06:08:02.405064+0000","last_undegraded":"2023-11-24T06:08:02.405064+0000","last_fullsized":"2023-11-24T06:08:02.405064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:11.429196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858885+0000","last_change":"2023-11-24T06:08:01.441968+0000","last_active":"2023-11-24T06:08:02.858885+0000","last_peered":"2023-11-24T06:08:02.858885+0000","last_clean":"2023-11-24T06:08:02.858885+0000","last_became_active":"2023-11-24T06:08:01.440920+0000","last_became_peered":"2023-11-24T06:08:01.440920+0000","last_unstale":"2023-11-24T06:08:02.858885+0000","last_undegraded":"2023-11-24T06:08:02.858885+0000","last_fullsized":"2023-11-24T06:08:02.858885+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:02:53.563916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.404769+0000","last_change":"2023-11-24T06:08:00.423440+0000","last_active":"2023-11-24T06:08:02.404769+0000","last_peered":"2023-11-24T06:08:02.404769+0000","last_clean":"2023-11-24T06:08:02.404769+0000","last_became_active":"2023-11-24T06:08:00.423175+0000","last_became_peered":"2023-11-24T06:08:00.423175+0000","last_unstale":"2023-11-24T06:08:02.404769+0000","last_undegraded":"2023-11-24T06:08:02.404769+0000","last_fullsized":"2023-11-24T06:08:02.404769+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:35:39.360506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859329+0000","last_change":"2023-11-24T06:08:01.435280+0000","last_active":"2023-11-24T06:08:02.859329+0000","last_peered":"2023-11-24T06:08:02.859329+0000","last_clean":"2023-11-24T06:08:02.859329+0000","last_became_active":"2023-11-24T06:08:01.434963+0000","last_became_peered":"2023-11-24T06:08:01.434963+0000","last_unstale":"2023-11-24T06:08:02.859329+0000","last_undegraded":"2023-11-24T06:08:02.859329+0000","last_fullsized":"2023-11-24T06:08:02.859329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:38:12.787229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.401899+0000","last_change":"2023-11-24T06:08:00.420129+0000","last_active":"2023-11-24T06:08:02.401899+0000","last_peered":"2023-11-24T06:08:02.401899+0000","last_clean":"2023-11-24T06:08:02.401899+0000","last_became_active":"2023-11-24T06:08:00.418612+0000","last_became_peered":"2023-11-24T06:08:00.418612+0000","last_unstale":"2023-11-24T06:08:02.401899+0000","last_undegraded":"2023-11-24T06:08:02.401899+0000","last_fullsized":"2023-11-24T06:08:02.401899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:40:19.498676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859549+0000","last_change":"2023-11-24T06:08:01.430171+0000","last_active":"2023-11-24T06:08:02.859549+0000","last_peered":"2023-11-24T06:08:02.859549+0000","last_clean":"2023-11-24T06:08:02.859549+0000","last_became_active":"2023-11-24T06:08:01.428562+0000","last_became_peered":"2023-11-24T06:08:01.428562+0000","last_unstale":"2023-11-24T06:08:02.859549+0000","last_undegraded":"2023-11-24T06:08:02.859549+0000","last_fullsized":"2023-11-24T06:08:02.859549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:29.649298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.428323+0000","last_change":"2023-11-24T06:08:01.433086+0000","last_active":"2023-11-24T06:08:02.428323+0000","last_peered":"2023-11-24T06:08:02.428323+0000","last_clean":"2023-11-24T06:08:02.428323+0000","last_became_active":"2023-11-24T06:08:01.432747+0000","last_became_peered":"2023-11-24T06:08:01.432747+0000","last_unstale":"2023-11-24T06:08:02.428323+0000","last_undegraded":"2023-11-24T06:08:02.428323+0000","last_fullsized":"2023-11-24T06:08:02.428323+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:59.812992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.407725+0000","last_change":"2023-11-24T06:08:00.416138+0000","last_active":"2023-11-24T06:08:02.407725+0000","last_peered":"2023-11-24T06:08:02.407725+0000","last_clean":"2023-11-24T06:08:02.407725+0000","last_became_active":"2023-11-24T06:08:00.415909+0000","last_became_peered":"2023-11-24T06:08:00.415909+0000","last_unstale":"2023-11-24T06:08:02.407725+0000","last_undegraded":"2023-11-24T06:08:02.407725+0000","last_fullsized":"2023-11-24T06:08:02.407725+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:05:31.619293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858173+0000","last_change":"2023-11-24T06:08:01.438377+0000","last_active":"2023-11-24T06:08:02.858173+0000","last_peered":"2023-11-24T06:08:02.858173+0000","last_clean":"2023-11-24T06:08:02.858173+0000","last_became_active":"2023-11-24T06:08:01.438142+0000","last_became_peered":"2023-11-24T06:08:01.438142+0000","last_unstale":"2023-11-24T06:08:02.858173+0000","last_undegraded":"2023-11-24T06:08:02.858173+0000","last_fullsized":"2023-11-24T06:08:02.858173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:48:22.215190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.401927+0000","last_change":"2023-11-24T06:08:00.419708+0000","last_active":"2023-11-24T06:08:02.401927+0000","last_peered":"2023-11-24T06:08:02.401927+0000","last_clean":"2023-11-24T06:08:02.401927+0000","last_became_active":"2023-11-24T06:08:00.419444+0000","last_became_peered":"2023-11-24T06:08:00.419444+0000","last_unstale":"2023-11-24T06:08:02.401927+0000","last_undegraded":"2023-11-24T06:08:02.401927+0000","last_fullsized":"2023-11-24T06:08:02.401927+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:39:13.152063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:01.425349+0000","last_change":"2023-11-24T06:07:49.463719+0000","last_active":"2023-11-24T06:08:01.425349+0000","last_peered":"2023-11-24T06:08:01.425349+0000","last_clean":"2023-11-24T06:08:01.425349+0000","last_became_active":"2023-11-24T06:07:49.462736+0000","last_became_peered":"2023-11-24T06:07:49.462736+0000","last_unstale":"2023-11-24T06:08:01.425349+0000","last_undegraded":"2023-11-24T06:08:01.425349+0000","last_fullsized":"2023-11-24T06:08:01.425349+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_clean_scrub_stamp":"2023-11-24T06:07:47.860604+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:16.508770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.857710+0000","last_change":"2023-11-24T06:08:01.438231+0000","last_active":"2023-11-24T06:08:02.857710+0000","last_peered":"2023-11-24T06:08:02.857710+0000","last_clean":"2023-11-24T06:08:02.857710+0000","last_became_active":"2023-11-24T06:08:01.437970+0000","last_became_peered":"2023-11-24T06:08:01.437970+0000","last_unstale":"2023-11-24T06:08:02.857710+0000","last_undegraded":"2023-11-24T06:08:02.857710+0000","last_fullsized":"2023-11-24T06:08:02.857710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:04.385452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.405157+0000","last_change":"2023-11-24T06:08:00.423316+0000","last_active":"2023-11-24T06:08:02.405157+0000","last_peered":"2023-11-24T06:08:02.405157+0000","last_clean":"2023-11-24T06:08:02.405157+0000","last_became_active":"2023-11-24T06:08:00.423046+0000","last_became_peered":"2023-11-24T06:08:00.423046+0000","last_unstale":"2023-11-24T06:08:02.405157+0000","last_undegraded":"2023-11-24T06:08:02.405157+0000","last_fullsized":"2023-11-24T06:08:02.405157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:17.143751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.405285+0000","last_change":"2023-11-24T06:08:00.424803+0000","last_active":"2023-11-24T06:08:02.405285+0000","last_peered":"2023-11-24T06:08:02.405285+0000","last_clean":"2023-11-24T06:08:02.405285+0000","last_became_active":"2023-11-24T06:08:00.424473+0000","last_became_peered":"2023-11-24T06:08:00.424473+0000","last_unstale":"2023-11-24T06:08:02.405285+0000","last_undegraded":"2023-11-24T06:08:02.405285+0000","last_fullsized":"2023-11-24T06:08:02.405285+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:00.828006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.428618+0000","last_change":"2023-11-24T06:08:01.432093+0000","last_active":"2023-11-24T06:08:02.428618+0000","last_peered":"2023-11-24T06:08:02.428618+0000","last_clean":"2023-11-24T06:08:02.428618+0000","last_became_active":"2023-11-24T06:08:01.431795+0000","last_became_peered":"2023-11-24T06:08:01.431795+0000","last_unstale":"2023-11-24T06:08:02.428618+0000","last_undegraded":"2023-11-24T06:08:02.428618+0000","last_fullsized":"2023-11-24T06:08:02.428618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:47:16.508162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.428443+0000","last_change":"2023-11-24T06:08:00.416062+0000","last_active":"2023-11-24T06:08:02.428443+0000","last_peered":"2023-11-24T06:08:02.428443+0000","last_clean":"2023-11-24T06:08:02.428443+0000","last_became_active":"2023-11-24T06:08:00.415783+0000","last_became_peered":"2023-11-24T06:08:00.415783+0000","last_unstale":"2023-11-24T06:08:02.428443+0000","last_undegraded":"2023-11-24T06:08:02.428443+0000","last_fullsized":"2023-11-24T06:08:02.428443+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:57:03.380514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858401+0000","last_change":"2023-11-24T06:08:01.437911+0000","last_active":"2023-11-24T06:08:02.858401+0000","last_peered":"2023-11-24T06:08:02.858401+0000","last_clean":"2023-11-24T06:08:02.858401+0000","last_became_active":"2023-11-24T06:08:01.437544+0000","last_became_peered":"2023-11-24T06:08:01.437544+0000","last_unstale":"2023-11-24T06:08:02.858401+0000","last_undegraded":"2023-11-24T06:08:02.858401+0000","last_fullsized":"2023-11-24T06:08:02.858401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:02:53.563916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.402168+0000","last_change":"2023-11-24T06:08:00.421447+0000","last_active":"2023-11-24T06:08:02.402168+0000","last_peered":"2023-11-24T06:08:02.402168+0000","last_clean":"2023-11-24T06:08:02.402168+0000","last_became_active":"2023-11-24T06:08:00.421145+0000","last_became_peered":"2023-11-24T06:08:00.421145+0000","last_unstale":"2023-11-24T06:08:02.402168+0000","last_undegraded":"2023-11-24T06:08:02.402168+0000","last_fullsized":"2023-11-24T06:08:02.402168+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:29.195643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858415+0000","last_change":"2023-11-24T06:08:01.427774+0000","last_active":"2023-11-24T06:08:02.858415+0000","last_peered":"2023-11-24T06:08:02.858415+0000","last_clean":"2023-11-24T06:08:02.858415+0000","last_became_active":"2023-11-24T06:08:01.427435+0000","last_became_peered":"2023-11-24T06:08:01.427435+0000","last_unstale":"2023-11-24T06:08:02.858415+0000","last_undegraded":"2023-11-24T06:08:02.858415+0000","last_fullsized":"2023-11-24T06:08:02.858415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:28.564398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.405130+0000","last_change":"2023-11-24T06:08:00.424213+0000","last_active":"2023-11-24T06:08:02.405130+0000","last_peered":"2023-11-24T06:08:02.405130+0000","last_clean":"2023-11-24T06:08:02.405130+0000","last_became_active":"2023-11-24T06:08:00.423943+0000","last_became_peered":"2023-11-24T06:08:00.423943+0000","last_unstale":"2023-11-24T06:08:02.405130+0000","last_undegraded":"2023-11-24T06:08:02.405130+0000","last_fullsized":"2023-11-24T06:08:02.405130+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:15.532537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858798+0000","last_change":"2023-11-24T06:08:01.420581+0000","last_active":"2023-11-24T06:08:02.858798+0000","last_peered":"2023-11-24T06:08:02.858798+0000","last_clean":"2023-11-24T06:08:02.858798+0000","last_became_active":"2023-11-24T06:08:01.420103+0000","last_became_peered":"2023-11-24T06:08:01.420103+0000","last_unstale":"2023-11-24T06:08:02.858798+0000","last_undegraded":"2023-11-24T06:08:02.858798+0000","last_fullsized":"2023-11-24T06:08:02.858798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:42.556322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.404691+0000","last_change":"2023-11-24T06:08:00.420593+0000","last_active":"2023-11-24T06:08:02.404691+0000","last_peered":"2023-11-24T06:08:02.404691+0000","last_clean":"2023-11-24T06:08:02.404691+0000","last_became_active":"2023-11-24T06:08:00.420286+0000","last_became_peered":"2023-11-24T06:08:00.420286+0000","last_unstale":"2023-11-24T06:08:02.404691+0000","last_undegraded":"2023-11-24T06:08:02.404691+0000","last_fullsized":"2023-11-24T06:08:02.404691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T06:14:28.644360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858190+0000","last_change":"2023-11-24T06:08:01.427055+0000","last_active":"2023-11-24T06:08:02.858190+0000","last_peered":"2023-11-24T06:08:02.858190+0000","last_clean":"2023-11-24T06:08:02.858190+0000","last_became_active":"2023-11-24T06:08:01.426823+0000","last_became_peered":"2023-11-24T06:08:01.426823+0000","last_unstale":"2023-11-24T06:08:02.858190+0000","last_undegraded":"2023-11-24T06:08:02.858190+0000","last_fullsized":"2023-11-24T06:08:02.858190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:06:58.673664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.404734+0000","last_change":"2023-11-24T06:08:00.421597+0000","last_active":"2023-11-24T06:08:02.404734+0000","last_peered":"2023-11-24T06:08:02.404734+0000","last_clean":"2023-11-24T06:08:02.404734+0000","last_became_active":"2023-11-24T06:08:00.421316+0000","last_became_peered":"2023-11-24T06:08:00.421316+0000","last_unstale":"2023-11-24T06:08:02.404734+0000","last_undegraded":"2023-11-24T06:08:02.404734+0000","last_fullsized":"2023-11-24T06:08:02.404734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:06:17.143751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858069+0000","last_change":"2023-11-24T06:08:01.430666+0000","last_active":"2023-11-24T06:08:02.858069+0000","last_peered":"2023-11-24T06:08:02.858069+0000","last_clean":"2023-11-24T06:08:02.858069+0000","last_became_active":"2023-11-24T06:08:01.430401+0000","last_became_peered":"2023-11-24T06:08:01.430401+0000","last_unstale":"2023-11-24T06:08:02.858069+0000","last_undegraded":"2023-11-24T06:08:02.858069+0000","last_fullsized":"2023-11-24T06:08:02.858069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:47:16.508162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.405217+0000","last_change":"2023-11-24T06:08:00.424291+0000","last_active":"2023-11-24T06:08:02.405217+0000","last_peered":"2023-11-24T06:08:02.405217+0000","last_clean":"2023-11-24T06:08:02.405217+0000","last_became_active":"2023-11-24T06:08:00.422721+0000","last_became_peered":"2023-11-24T06:08:00.422721+0000","last_unstale":"2023-11-24T06:08:02.405217+0000","last_undegraded":"2023-11-24T06:08:02.405217+0000","last_fullsized":"2023-11-24T06:08:02.405217+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:29:29.195643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.427283+0000","last_change":"2023-11-24T06:08:01.424141+0000","last_active":"2023-11-24T06:08:02.427283+0000","last_peered":"2023-11-24T06:08:02.427283+0000","last_clean":"2023-11-24T06:08:02.427283+0000","last_became_active":"2023-11-24T06:08:01.423837+0000","last_became_peered":"2023-11-24T06:08:01.423837+0000","last_unstale":"2023-11-24T06:08:02.427283+0000","last_undegraded":"2023-11-24T06:08:02.427283+0000","last_fullsized":"2023-11-24T06:08:02.427283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:40:20.503614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.407768+0000","last_change":"2023-11-24T06:08:00.420556+0000","last_active":"2023-11-24T06:08:02.407768+0000","last_peered":"2023-11-24T06:08:02.407768+0000","last_clean":"2023-11-24T06:08:02.407768+0000","last_became_active":"2023-11-24T06:08:00.420250+0000","last_became_peered":"2023-11-24T06:08:00.420250+0000","last_unstale":"2023-11-24T06:08:02.407768+0000","last_undegraded":"2023-11-24T06:08:02.407768+0000","last_fullsized":"2023-11-24T06:08:02.407768+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:48:11.429196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859838+0000","last_change":"2023-11-24T06:08:01.440217+0000","last_active":"2023-11-24T06:08:02.859838+0000","last_peered":"2023-11-24T06:08:02.859838+0000","last_clean":"2023-11-24T06:08:02.859838+0000","last_became_active":"2023-11-24T06:08:01.439942+0000","last_became_peered":"2023-11-24T06:08:01.439942+0000","last_unstale":"2023-11-24T06:08:02.859838+0000","last_undegraded":"2023-11-24T06:08:02.859838+0000","last_fullsized":"2023-11-24T06:08:02.859838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:26:20.272046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.404803+0000","last_change":"2023-11-24T06:08:00.413735+0000","last_active":"2023-11-24T06:08:02.404803+0000","last_peered":"2023-11-24T06:08:02.404803+0000","last_clean":"2023-11-24T06:08:02.404803+0000","last_became_active":"2023-11-24T06:08:00.413375+0000","last_became_peered":"2023-11-24T06:08:00.413375+0000","last_unstale":"2023-11-24T06:08:02.404803+0000","last_undegraded":"2023-11-24T06:08:02.404803+0000","last_fullsized":"2023-11-24T06:08:02.404803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:33:20.079860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858677+0000","last_change":"2023-11-24T06:08:01.435772+0000","last_active":"2023-11-24T06:08:02.858677+0000","last_peered":"2023-11-24T06:08:02.858677+0000","last_clean":"2023-11-24T06:08:02.858677+0000","last_became_active":"2023-11-24T06:08:01.435495+0000","last_became_peered":"2023-11-24T06:08:01.435495+0000","last_unstale":"2023-11-24T06:08:02.858677+0000","last_undegraded":"2023-11-24T06:08:02.858677+0000","last_fullsized":"2023-11-24T06:08:02.858677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:01.832945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.407811+0000","last_change":"2023-11-24T06:08:00.415129+0000","last_active":"2023-11-24T06:08:02.407811+0000","last_peered":"2023-11-24T06:08:02.407811+0000","last_clean":"2023-11-24T06:08:02.407811+0000","last_became_active":"2023-11-24T06:08:00.414888+0000","last_became_peered":"2023-11-24T06:08:00.414888+0000","last_unstale":"2023-11-24T06:08:02.407811+0000","last_undegraded":"2023-11-24T06:08:02.407811+0000","last_fullsized":"2023-11-24T06:08:02.407811+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:59.395455+0000","last_clean_scrub_stamp":"2023-11-24T06:07:59.395455+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:25:41.551384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859426+0000","last_change":"2023-11-24T06:08:01.442379+0000","last_active":"2023-11-24T06:08:02.859426+0000","last_peered":"2023-11-24T06:08:02.859426+0000","last_clean":"2023-11-24T06:08:02.859426+0000","last_became_active":"2023-11-24T06:08:01.442109+0000","last_became_peered":"2023-11-24T06:08:01.442109+0000","last_unstale":"2023-11-24T06:08:02.859426+0000","last_undegraded":"2023-11-24T06:08:02.859426+0000","last_fullsized":"2023-11-24T06:08:02.859426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:19:13.981084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.427230+0000","last_change":"2023-11-24T06:08:01.429866+0000","last_active":"2023-11-24T06:08:02.427230+0000","last_peered":"2023-11-24T06:08:02.427230+0000","last_clean":"2023-11-24T06:08:02.427230+0000","last_became_active":"2023-11-24T06:08:01.429446+0000","last_became_peered":"2023-11-24T06:08:01.429446+0000","last_unstale":"2023-11-24T06:08:02.427230+0000","last_undegraded":"2023-11-24T06:08:02.427230+0000","last_fullsized":"2023-11-24T06:08:02.427230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:52:57.033291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.429315+0000","last_change":"2023-11-24T06:08:01.435709+0000","last_active":"2023-11-24T06:08:02.429315+0000","last_peered":"2023-11-24T06:08:02.429315+0000","last_clean":"2023-11-24T06:08:02.429315+0000","last_became_active":"2023-11-24T06:08:01.435488+0000","last_became_peered":"2023-11-24T06:08:01.435488+0000","last_unstale":"2023-11-24T06:08:02.429315+0000","last_undegraded":"2023-11-24T06:08:02.429315+0000","last_fullsized":"2023-11-24T06:08:02.429315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:29:16.537475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859231+0000","last_change":"2023-11-24T06:08:01.434372+0000","last_active":"2023-11-24T06:08:02.859231+0000","last_peered":"2023-11-24T06:08:02.859231+0000","last_clean":"2023-11-24T06:08:02.859231+0000","last_became_active":"2023-11-24T06:08:01.433776+0000","last_became_peered":"2023-11-24T06:08:01.433776+0000","last_unstale":"2023-11-24T06:08:02.859231+0000","last_undegraded":"2023-11-24T06:08:02.859231+0000","last_fullsized":"2023-11-24T06:08:02.859231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:55:45.332918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858468+0000","last_change":"2023-11-24T06:08:01.433754+0000","last_active":"2023-11-24T06:08:02.858468+0000","last_peered":"2023-11-24T06:08:02.858468+0000","last_clean":"2023-11-24T06:08:02.858468+0000","last_became_active":"2023-11-24T06:08:01.433473+0000","last_became_peered":"2023-11-24T06:08:01.433473+0000","last_unstale":"2023-11-24T06:08:02.858468+0000","last_undegraded":"2023-11-24T06:08:02.858468+0000","last_fullsized":"2023-11-24T06:08:02.858468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:47:16.508162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858224+0000","last_change":"2023-11-24T06:08:01.428520+0000","last_active":"2023-11-24T06:08:02.858224+0000","last_peered":"2023-11-24T06:08:02.858224+0000","last_clean":"2023-11-24T06:08:02.858224+0000","last_became_active":"2023-11-24T06:08:01.428236+0000","last_became_peered":"2023-11-24T06:08:01.428236+0000","last_unstale":"2023-11-24T06:08:02.858224+0000","last_undegraded":"2023-11-24T06:08:02.858224+0000","last_fullsized":"2023-11-24T06:08:02.858224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T18:05:01.832945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.428655+0000","last_change":"2023-11-24T06:08:01.433972+0000","last_active":"2023-11-24T06:08:02.428655+0000","last_peered":"2023-11-24T06:08:02.428655+0000","last_clean":"2023-11-24T06:08:02.428655+0000","last_became_active":"2023-11-24T06:08:01.433685+0000","last_became_peered":"2023-11-24T06:08:01.433685+0000","last_unstale":"2023-11-24T06:08:02.428655+0000","last_undegraded":"2023-11-24T06:08:02.428655+0000","last_fullsized":"2023-11-24T06:08:02.428655+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:52.422428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.427720+0000","last_change":"2023-11-24T06:08:01.435802+0000","last_active":"2023-11-24T06:08:02.427720+0000","last_peered":"2023-11-24T06:08:02.427720+0000","last_clean":"2023-11-24T06:08:02.427720+0000","last_became_active":"2023-11-24T06:08:01.435228+0000","last_became_peered":"2023-11-24T06:08:01.435228+0000","last_unstale":"2023-11-24T06:08:02.427720+0000","last_undegraded":"2023-11-24T06:08:02.427720+0000","last_fullsized":"2023-11-24T06:08:02.427720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:52.422428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858155+0000","last_change":"2023-11-24T06:08:01.432295+0000","last_active":"2023-11-24T06:08:02.858155+0000","last_peered":"2023-11-24T06:08:02.858155+0000","last_clean":"2023-11-24T06:08:02.858155+0000","last_became_active":"2023-11-24T06:08:01.431767+0000","last_became_peered":"2023-11-24T06:08:01.431767+0000","last_unstale":"2023-11-24T06:08:02.858155+0000","last_undegraded":"2023-11-24T06:08:02.858155+0000","last_fullsized":"2023-11-24T06:08:02.858155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:52.422428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858440+0000","last_change":"2023-11-24T06:08:01.434132+0000","last_active":"2023-11-24T06:08:02.858440+0000","last_peered":"2023-11-24T06:08:02.858440+0000","last_clean":"2023-11-24T06:08:02.858440+0000","last_became_active":"2023-11-24T06:08:01.433678+0000","last_became_peered":"2023-11-24T06:08:01.433678+0000","last_unstale":"2023-11-24T06:08:02.858440+0000","last_undegraded":"2023-11-24T06:08:02.858440+0000","last_fullsized":"2023-11-24T06:08:02.858440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:28:52.422428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.857901+0000","last_change":"2023-11-24T06:08:01.424409+0000","last_active":"2023-11-24T06:08:02.857901+0000","last_peered":"2023-11-24T06:08:02.857901+0000","last_clean":"2023-11-24T06:08:02.857901+0000","last_became_active":"2023-11-24T06:08:01.424152+0000","last_became_peered":"2023-11-24T06:08:01.424152+0000","last_unstale":"2023-11-24T06:08:02.857901+0000","last_undegraded":"2023-11-24T06:08:02.857901+0000","last_fullsized":"2023-11-24T06:08:02.857901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:40:20.503614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.859100+0000","last_change":"2023-11-24T06:08:01.436895+0000","last_active":"2023-11-24T06:08:02.859100+0000","last_peered":"2023-11-24T06:08:02.859100+0000","last_clean":"2023-11-24T06:08:02.859100+0000","last_became_active":"2023-11-24T06:08:01.436631+0000","last_became_peered":"2023-11-24T06:08:01.436631+0000","last_unstale":"2023-11-24T06:08:02.859100+0000","last_undegraded":"2023-11-24T06:08:02.859100+0000","last_fullsized":"2023-11-24T06:08:02.859100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:32:41.812446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.858334+0000","last_change":"2023-11-24T06:08:01.435442+0000","last_active":"2023-11-24T06:08:02.858334+0000","last_peered":"2023-11-24T06:08:02.858334+0000","last_clean":"2023-11-24T06:08:02.858334+0000","last_became_active":"2023-11-24T06:08:01.435135+0000","last_became_peered":"2023-11-24T06:08:01.435135+0000","last_unstale":"2023-11-24T06:08:02.858334+0000","last_undegraded":"2023-11-24T06:08:02.858334+0000","last_fullsized":"2023-11-24T06:08:02.858334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:54:59.812992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-11-24T06:08:02.427573+0000","last_change":"2023-11-24T06:08:01.423313+0000","last_active":"2023-11-24T06:08:02.427573+0000","last_peered":"2023-11-24T06:08:02.427573+0000","last_clean":"2023-11-24T06:08:02.427573+0000","last_became_active":"2023-11-24T06:08:01.423042+0000","last_became_peered":"2023-11-24T06:08:01.423042+0000","last_unstale":"2023-11-24T06:08:02.427573+0000","last_undegraded":"2023-11-24T06:08:02.427573+0000","last_fullsized":"2023-11-24T06:08:02.427573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:00.400393+0000","last_clean_scrub_stamp":"2023-11-24T06:08:00.400393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:47:16.508162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":802816,"data_stored":661860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.7670000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9529999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.9640000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.048}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":661916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":72522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.309}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":71036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":71000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.476}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:08:10.866 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-24T06:08:10.866 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-24T06:08:10.866 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-24T06:08:10.866 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-24T06:08:11.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.016+0000 7f37ab884700 1 Processor -- start 2023-11-24T06:08:11.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.016+0000 7f37ab884700 1 -- start start 2023-11-24T06:08:11.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.017+0000 7f37ab884700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f37a406b620 0x7f37a40b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.017+0000 7f37ab884700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f37a40b5ab0 0x7f37a40b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.017+0000 7f37ab884700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a4071f50 con 0x7f37a41478c0 2023-11-24T06:08:11.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.017+0000 7f37ab884700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a4072230 con 0x7f37a40b5ab0 2023-11-24T06:08:11.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.017+0000 7f37ab884700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a4072510 con 0x7f37a406b620 2023-11-24T06:08:11.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.017+0000 7f37a8e1f700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f37a40b5ab0 0x7f37a40b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.017+0000 7f37a9e21700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f37a41478c0 0x7f37a4161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53156/0 (socket says 172.21.15.91:53156) 2023-11-24T06:08:11.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.017+0000 7f37a9e21700 1 -- 172.21.15.91:0/3912968437 learned_addr learned my addr 172.21.15.91:0/3912968437 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:11.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.018+0000 7f37a9620700 1 --2- 172.21.15.91:0/3912968437 >> v2:172.21.15.94:3300/0 conn(0x7f37a406b620 0x7f37a40b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.018+0000 7f37a8e1f700 1 -- 172.21.15.91:0/3912968437 >> v2:172.21.15.94:3300/0 conn(0x7f37a406b620 msgr2=0x7f37a40b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:11.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.018+0000 7f37a8e1f700 1 --2- 172.21.15.91:0/3912968437 >> v2:172.21.15.94:3300/0 conn(0x7f37a406b620 0x7f37a40b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.018+0000 7f37a8e1f700 1 -- 172.21.15.91:0/3912968437 >> v1:172.21.15.91:6789/0 conn(0x7f37a41478c0 legacy=0x7f37a4161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.018+0000 7f37a8e1f700 1 -- 172.21.15.91:0/3912968437 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a40727f0 con 0x7f37a40b5ab0 2023-11-24T06:08:11.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.018+0000 7f379bfff700 1 -- 172.21.15.91:0/3912968437 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653756174 0 0) 0x7f37a4071f50 con 0x7f37a41478c0 2023-11-24T06:08:11.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.018+0000 7f37a8e1f700 1 --2- 172.21.15.91:0/3912968437 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f37a40b5ab0 0x7f37a40b3f80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f37a406d2e0 tx=0x7f37a0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42778eec64e0812 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:11.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.018+0000 7f379bfff700 1 -- 172.21.15.91:0/3912968437 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a0011830 con 0x7f37a40b5ab0 2023-11-24T06:08:11.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.018+0000 7f379bfff700 1 -- 172.21.15.91:0/3912968437 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a0011990 con 0x7f37a40b5ab0 2023-11-24T06:08:11.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.018+0000 7f379bfff700 1 -- 172.21.15.91:0/3912968437 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a0011d20 con 0x7f37a40b5ab0 2023-11-24T06:08:11.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.018+0000 7f37ab884700 1 -- 172.21.15.91:0/3912968437 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f37a40b5ab0 msgr2=0x7f37a40b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.018+0000 7f37ab884700 1 --2- 172.21.15.91:0/3912968437 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f37a40b5ab0 0x7f37a40b3f80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f37a406d2e0 tx=0x7f37a0009b20 comp rx=0 tx=0).stop 2023-11-24T06:08:11.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.018+0000 7f37ab884700 1 -- 172.21.15.91:0/3912968437 shutdown_connections 2023-11-24T06:08:11.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.018+0000 7f37ab884700 1 --2- 172.21.15.91:0/3912968437 >> v2:172.21.15.94:3300/0 conn(0x7f37a406b620 0x7f37a40b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.018+0000 7f37ab884700 1 --2- 172.21.15.91:0/3912968437 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f37a40b5ab0 0x7f37a40b3f80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.018+0000 7f37ab884700 1 -- 172.21.15.91:0/3912968437 >> 172.21.15.91:0/3912968437 conn(0x7f37a40b2740 msgr2=0x7f37a40b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:11.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.019+0000 7f37ab884700 1 -- 172.21.15.91:0/3912968437 shutdown_connections 2023-11-24T06:08:11.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.019+0000 7f37ab884700 1 -- 172.21.15.91:0/3912968437 wait complete. 2023-11-24T06:08:11.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.020+0000 7f37ab884700 1 Processor -- start 2023-11-24T06:08:11.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.020+0000 7f37ab884700 1 -- start start 2023-11-24T06:08:11.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.020+0000 7f37ab884700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f37a406b620 0x7f37a41e5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.020+0000 7f37ab884700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f37a40b5ab0 0x7f37a41f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.020+0000 7f37ab884700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f37a41478c0 0x7f37a41f7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.020+0000 7f37ab884700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a40c72d0 con 0x7f37a406b620 2023-11-24T06:08:11.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.020+0000 7f37ab884700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a40c66c0 con 0x7f37a41478c0 2023-11-24T06:08:11.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.020+0000 7f37ab884700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a40c69a0 con 0x7f37a40b5ab0 2023-11-24T06:08:11.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f37a9620700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f37a406b620 0x7f37a41e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f37a9620700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f37a406b620 0x7f37a41e5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58224/0 (socket says 172.21.15.91:58224) 2023-11-24T06:08:11.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f37a9620700 1 -- 172.21.15.91:0/1097614787 learned_addr learned my addr 172.21.15.91:0/1097614787 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:11.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f37a9e21700 1 --2- 172.21.15.91:0/1097614787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f37a41478c0 0x7f37a41f7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f37a8e1f700 1 --2- 172.21.15.91:0/1097614787 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f37a40b5ab0 0x7f37a41f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f37a9620700 1 -- 172.21.15.91:0/1097614787 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f37a40b5ab0 msgr2=0x7f37a41f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f37a9620700 1 --2- 172.21.15.91:0/1097614787 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f37a40b5ab0 0x7f37a41f7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f37a9620700 1 -- 172.21.15.91:0/1097614787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f37a41478c0 msgr2=0x7f37a41f7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f37a9620700 1 --2- 172.21.15.91:0/1097614787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f37a41478c0 0x7f37a41f7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f37a9620700 1 -- 172.21.15.91:0/1097614787 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a00094a0 con 0x7f37a406b620 2023-11-24T06:08:11.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f37a9e21700 1 --2- 172.21.15.91:0/1097614787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f37a41478c0 0x7f37a41f7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:11.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f37a9620700 1 --2- 172.21.15.91:0/1097614787 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f37a406b620 0x7f37a41e5260 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f379400c7f0 tx=0x7f379400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:11.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f3799ffb700 1 -- 172.21.15.91:0/1097614787 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3794011800 con 0x7f37a406b620 2023-11-24T06:08:11.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f3799ffb700 1 -- 172.21.15.91:0/1097614787 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3794011960 con 0x7f37a406b620 2023-11-24T06:08:11.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f37ab884700 1 -- 172.21.15.91:0/1097614787 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a40c6c80 con 0x7f37a406b620 2023-11-24T06:08:11.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f3799ffb700 1 -- 172.21.15.91:0/1097614787 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3794011cf0 con 0x7f37a406b620 2023-11-24T06:08:11.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.021+0000 7f37ab884700 1 -- 172.21.15.91:0/1097614787 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a41e57a0 con 0x7f37a406b620 2023-11-24T06:08:11.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.022+0000 7f37ab884700 1 -- 172.21.15.91:0/1097614787 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a4068670 con 0x7f37a406b620 2023-11-24T06:08:11.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.023+0000 7f3799ffb700 1 -- 172.21.15.91:0/1097614787 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3794018360 con 0x7f37a406b620 2023-11-24T06:08:11.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.023+0000 7f3799ffb700 1 --2- 172.21.15.91:0/1097614787 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f378806aed0 0x7f378806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.023+0000 7f3799ffb700 1 -- 172.21.15.91:0/1097614787 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37940895e0 con 0x7f37a406b620 2023-11-24T06:08:11.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.023+0000 7f37a8e1f700 1 --2- 172.21.15.91:0/1097614787 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f378806aed0 0x7f378806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.031 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.023+0000 7f37a8e1f700 1 --2- 172.21.15.91:0/1097614787 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f378806aed0 0x7f378806d390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f37a40b5f10 tx=0x7f37a0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:11.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.028+0000 7f3799ffb700 1 -- 172.21.15.91:0/1097614787 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f37940546a0 con 0x7f37a406b620 2023-11-24T06:08:11.268 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.263+0000 7f37ab884700 1 -- 172.21.15.91:0/1097614787 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f37a40b42a0 con 0x7f37a406b620 2023-11-24T06:08:11.268 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.264+0000 7f3799ffb700 1 -- 172.21.15.91:0/1097614787 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f3794057cc0 con 0x7f37a406b620 2023-11-24T06:08:11.268 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:11.268 INFO:teuthology.orchestra.run.smithi091.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-24T06:08:11.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.266+0000 7f37ab884700 1 -- 172.21.15.91:0/1097614787 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f378806aed0 msgr2=0x7f378806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.266+0000 7f37ab884700 1 --2- 172.21.15.91:0/1097614787 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f378806aed0 0x7f378806d390 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f37a40b5f10 tx=0x7f37a0008030 comp rx=0 tx=0).stop 2023-11-24T06:08:11.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.266+0000 7f37ab884700 1 -- 172.21.15.91:0/1097614787 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f37a406b620 msgr2=0x7f37a41e5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.266+0000 7f37ab884700 1 --2- 172.21.15.91:0/1097614787 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f37a406b620 0x7f37a41e5260 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f379400c7f0 tx=0x7f379400cbb0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.266+0000 7f37ab884700 1 -- 172.21.15.91:0/1097614787 shutdown_connections 2023-11-24T06:08:11.271 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.266+0000 7f37ab884700 1 --2- 172.21.15.91:0/1097614787 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f378806aed0 0x7f378806d390 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.271 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.266+0000 7f37ab884700 1 --2- 172.21.15.91:0/1097614787 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f37a41478c0 0x7f37a41f7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.271 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.266+0000 7f37ab884700 1 --2- 172.21.15.91:0/1097614787 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f37a406b620 0x7f37a41e5260 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.271 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.266+0000 7f37ab884700 1 --2- 172.21.15.91:0/1097614787 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f37a40b5ab0 0x7f37a41f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.271 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.266+0000 7f37ab884700 1 -- 172.21.15.91:0/1097614787 >> 172.21.15.91:0/1097614787 conn(0x7f37a40b2740 msgr2=0x7f37a41464d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:11.271 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.266+0000 7f37ab884700 1 -- 172.21.15.91:0/1097614787 shutdown_connections 2023-11-24T06:08:11.271 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.266+0000 7f37ab884700 1 -- 172.21.15.91:0/1097614787 wait complete. 2023-11-24T06:08:11.282 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-24T06:08:11.282 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:11.402 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.399+0000 7fd180d9f700 1 Processor -- start 2023-11-24T06:08:11.403 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.399+0000 7fd180d9f700 1 -- start start 2023-11-24T06:08:11.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.399+0000 7fd180d9f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd17c06b620 0x7fd17c0b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.406 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd180d9f700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd17c06ee60 0x7fd17c0b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.406 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd180d9f700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17c071f50 con 0x7fd17c0b5ab0 2023-11-24T06:08:11.406 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd180d9f700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17c072230 con 0x7fd17c06ee60 2023-11-24T06:08:11.406 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd180d9f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c072510 con 0x7fd17c06b620 2023-11-24T06:08:11.406 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd17ad9d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd17c06ee60 0x7fd17c0b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.406 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd179d9b700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fd17c0b5ab0 0x7fd17c161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53166/0 (socket says 172.21.15.91:53166) 2023-11-24T06:08:11.406 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd179d9b700 1 -- 172.21.15.91:0/2406565738 learned_addr learned my addr 172.21.15.91:0/2406565738 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:11.406 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd17a59c700 1 --2- 172.21.15.91:0/2406565738 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd17c06b620 0x7fd17c0b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.406 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd17ad9d700 1 -- 172.21.15.91:0/2406565738 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd17c06b620 msgr2=0x7fd17c0b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:11.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd17ad9d700 1 --2- 172.21.15.91:0/2406565738 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd17c06b620 0x7fd17c0b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd17ad9d700 1 -- 172.21.15.91:0/2406565738 >> v1:172.21.15.91:6789/0 conn(0x7fd17c0b5ab0 legacy=0x7fd17c161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd17ad9d700 1 -- 172.21.15.91:0/2406565738 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c0727f0 con 0x7fd17c06ee60 2023-11-24T06:08:11.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd17959a700 1 -- 172.21.15.91:0/2406565738 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230410686 0 0) 0x7fd17c071f50 con 0x7fd17c0b5ab0 2023-11-24T06:08:11.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd17ad9d700 1 --2- 172.21.15.91:0/2406565738 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd17c06ee60 0x7fd17c0b3f80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd174009d90 tx=0x7fd17400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9025195ba6b58659 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:11.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd17959a700 1 -- 172.21.15.91:0/2406565738 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17400e070 con 0x7fd17c06ee60 2023-11-24T06:08:11.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd17959a700 1 -- 172.21.15.91:0/2406565738 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17400bbd0 con 0x7fd17c06ee60 2023-11-24T06:08:11.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.400+0000 7fd17959a700 1 -- 172.21.15.91:0/2406565738 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd174012430 con 0x7fd17c06ee60 2023-11-24T06:08:11.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.401+0000 7fd180d9f700 1 -- 172.21.15.91:0/2406565738 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd17c06ee60 msgr2=0x7fd17c0b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.401+0000 7fd180d9f700 1 --2- 172.21.15.91:0/2406565738 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd17c06ee60 0x7fd17c0b3f80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd174009d90 tx=0x7fd17400b330 comp rx=0 tx=0).stop 2023-11-24T06:08:11.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.401+0000 7fd180d9f700 1 -- 172.21.15.91:0/2406565738 shutdown_connections 2023-11-24T06:08:11.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.401+0000 7fd180d9f700 1 --2- 172.21.15.91:0/2406565738 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd17c06ee60 0x7fd17c0b3f80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.401+0000 7fd180d9f700 1 --2- 172.21.15.91:0/2406565738 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd17c06b620 0x7fd17c0b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.401+0000 7fd180d9f700 1 -- 172.21.15.91:0/2406565738 >> 172.21.15.91:0/2406565738 conn(0x7fd17c0b2740 msgr2=0x7fd17c0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:11.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.401+0000 7fd180d9f700 1 -- 172.21.15.91:0/2406565738 shutdown_connections 2023-11-24T06:08:11.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.401+0000 7fd180d9f700 1 -- 172.21.15.91:0/2406565738 wait complete. 2023-11-24T06:08:11.410 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.402+0000 7fd180d9f700 1 Processor -- start 2023-11-24T06:08:11.410 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.402+0000 7fd180d9f700 1 -- start start 2023-11-24T06:08:11.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd180d9f700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd17c06b620 0x7fd17c1e8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd180d9f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd17c06ee60 0x7fd17c0c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd180d9f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd17c0b5ab0 0x7fd17c1430d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd180d9f700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c0c6320 con 0x7fd17c06b620 2023-11-24T06:08:11.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd180d9f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c0c6600 con 0x7fd17c0b5ab0 2023-11-24T06:08:11.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd180d9f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17c0c68e0 con 0x7fd17c06ee60 2023-11-24T06:08:11.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd17a59c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd17c06b620 0x7fd17c1e8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd17a59c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd17c06b620 0x7fd17c1e8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58238/0 (socket says 172.21.15.91:58238) 2023-11-24T06:08:11.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd17a59c700 1 -- 172.21.15.91:0/1145380540 learned_addr learned my addr 172.21.15.91:0/1145380540 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:11.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd179d9b700 1 --2- 172.21.15.91:0/1145380540 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd17c06ee60 0x7fd17c0c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd17ad9d700 1 --2- 172.21.15.91:0/1145380540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd17c0b5ab0 0x7fd17c1430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd17a59c700 1 -- 172.21.15.91:0/1145380540 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd17c06ee60 msgr2=0x7fd17c0c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd17a59c700 1 --2- 172.21.15.91:0/1145380540 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd17c06ee60 0x7fd17c0c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd17a59c700 1 -- 172.21.15.91:0/1145380540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd17c0b5ab0 msgr2=0x7fd17c1430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd17a59c700 1 --2- 172.21.15.91:0/1145380540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd17c0b5ab0 0x7fd17c1430d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd17a59c700 1 -- 172.21.15.91:0/1145380540 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd174009a70 con 0x7fd17c06b620 2023-11-24T06:08:11.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd17ad9d700 1 --2- 172.21.15.91:0/1145380540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd17c0b5ab0 0x7fd17c1430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:11.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.403+0000 7fd17a59c700 1 --2- 172.21.15.91:0/1145380540 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd17c06b620 0x7fd17c1e8d00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd17000e810 tx=0x7fd17000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:11.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.404+0000 7fd16affd700 1 -- 172.21.15.91:0/1145380540 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd170013610 con 0x7fd17c06b620 2023-11-24T06:08:11.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.404+0000 7fd16affd700 1 -- 172.21.15.91:0/1145380540 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd170013770 con 0x7fd17c06b620 2023-11-24T06:08:11.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.404+0000 7fd180d9f700 1 -- 172.21.15.91:0/1145380540 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd17c0c6bc0 con 0x7fd17c06b620 2023-11-24T06:08:11.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.404+0000 7fd16affd700 1 -- 172.21.15.91:0/1145380540 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd170013b20 con 0x7fd17c06b620 2023-11-24T06:08:11.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.404+0000 7fd180d9f700 1 -- 172.21.15.91:0/1145380540 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd17c1e9420 con 0x7fd17c06b620 2023-11-24T06:08:11.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.404+0000 7fd180d9f700 1 -- 172.21.15.91:0/1145380540 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17c0ca130 con 0x7fd17c06b620 2023-11-24T06:08:11.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.405+0000 7fd16affd700 1 -- 172.21.15.91:0/1145380540 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd170013c80 con 0x7fd17c06b620 2023-11-24T06:08:11.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.405+0000 7fd16affd700 1 --2- 172.21.15.91:0/1145380540 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd15c06aed0 0x7fd15c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.405+0000 7fd179d9b700 1 --2- 172.21.15.91:0/1145380540 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd15c06aed0 0x7fd15c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.405+0000 7fd16affd700 1 -- 172.21.15.91:0/1145380540 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1700890a0 con 0x7fd17c06b620 2023-11-24T06:08:11.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.406+0000 7fd179d9b700 1 --2- 172.21.15.91:0/1145380540 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd15c06aed0 0x7fd15c06d390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd17c0cfc90 tx=0x7fd16c007e70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:11.414 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.410+0000 7fd16affd700 1 -- 172.21.15.91:0/1145380540 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd1700540c0 con 0x7fd17c06b620 2023-11-24T06:08:11.605 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.601+0000 7fd180d9f700 1 -- 172.21.15.91:0/1145380540 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd17c1e9420 con 0x7fd17c06b620 2023-11-24T06:08:11.606 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.602+0000 7fd16affd700 1 -- 172.21.15.91:0/1145380540 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7fd17c1e9420 con 0x7fd17c06b620 2023-11-24T06:08:11.606 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:11.606 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":20,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:02.415612+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:11.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.604+0000 7fd180d9f700 1 -- 172.21.15.91:0/1145380540 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd15c06aed0 msgr2=0x7fd15c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.604+0000 7fd180d9f700 1 --2- 172.21.15.91:0/1145380540 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd15c06aed0 0x7fd15c06d390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd17c0cfc90 tx=0x7fd16c007e70 comp rx=0 tx=0).stop 2023-11-24T06:08:11.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.604+0000 7fd180d9f700 1 -- 172.21.15.91:0/1145380540 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd17c06b620 msgr2=0x7fd17c1e8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.604+0000 7fd180d9f700 1 --2- 172.21.15.91:0/1145380540 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd17c06b620 0x7fd17c1e8d00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd17000e810 tx=0x7fd17000eb20 comp rx=0 tx=0).stop 2023-11-24T06:08:11.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.604+0000 7fd180d9f700 1 -- 172.21.15.91:0/1145380540 shutdown_connections 2023-11-24T06:08:11.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.604+0000 7fd180d9f700 1 --2- 172.21.15.91:0/1145380540 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd15c06aed0 0x7fd15c06d390 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.604+0000 7fd180d9f700 1 --2- 172.21.15.91:0/1145380540 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd17c0b5ab0 0x7fd17c1430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.604+0000 7fd180d9f700 1 --2- 172.21.15.91:0/1145380540 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd17c06b620 0x7fd17c1e8d00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.604+0000 7fd180d9f700 1 --2- 172.21.15.91:0/1145380540 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd17c06ee60 0x7fd17c0c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.604+0000 7fd180d9f700 1 -- 172.21.15.91:0/1145380540 >> 172.21.15.91:0/1145380540 conn(0x7fd17c0b2740 msgr2=0x7fd17c146e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:11.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.604+0000 7fd180d9f700 1 -- 172.21.15.91:0/1145380540 shutdown_connections 2023-11-24T06:08:11.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.604+0000 7fd180d9f700 1 -- 172.21.15.91:0/1145380540 wait complete. 2023-11-24T06:08:11.620 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:07:58.710106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:07:59.774819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:08:11.621 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:08:11.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.740+0000 7fb7c8139700 1 Processor -- start 2023-11-24T06:08:11.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.740+0000 7fb7c8139700 1 -- start start 2023-11-24T06:08:11.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.740+0000 7fb7c8139700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb7c015bde0 0x7fb7c015c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.740+0000 7fb7c8139700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fb7c00cf2b0 0x7fb7c00d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.741+0000 7fb7c8139700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c00d0a90 con 0x7fb7c00ce5c0 2023-11-24T06:08:11.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.741+0000 7fb7c8139700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c006ee10 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.741+0000 7fb7c8139700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c006f0f0 con 0x7fb7c015bde0 2023-11-24T06:08:11.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.741+0000 7fb7c56d4700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fb7c00cf2b0 0x7fb7c00d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.741+0000 7fb7c56d4700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fb7c00cf2b0 0x7fb7c00d01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58248/0 (socket says 172.21.15.91:58248) 2023-11-24T06:08:11.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.741+0000 7fb7c56d4700 1 -- 172.21.15.91:0/931634420 learned_addr learned my addr 172.21.15.91:0/931634420 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:11.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.741+0000 7fb7c5ed5700 1 --2- 172.21.15.91:0/931634420 >> v2:172.21.15.94:3300/0 conn(0x7fb7c015bde0 0x7fb7c015c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.741+0000 7fb7c56d4700 1 -- 172.21.15.91:0/931634420 >> v2:172.21.15.94:3300/0 conn(0x7fb7c015bde0 msgr2=0x7fb7c015c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:11.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.741+0000 7fb7c56d4700 1 --2- 172.21.15.91:0/931634420 >> v2:172.21.15.94:3300/0 conn(0x7fb7c015bde0 0x7fb7c015c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.741+0000 7fb7c56d4700 1 -- 172.21.15.91:0/931634420 >> v1:172.21.15.94:6789/0 conn(0x7fb7c00ce5c0 legacy=0x7fb7c00cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.741+0000 7fb7c56d4700 1 -- 172.21.15.91:0/931634420 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c006f3d0 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.741+0000 7fb7c56d4700 1 --2- 172.21.15.91:0/931634420 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7c00cf2b0 0x7fb7c00d01b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0072370 tx=0x7fb7b0009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da54f512963994 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:11.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.741+0000 7fb7c4ed3700 1 -- 172.21.15.91:0/931634420 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0011830 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.741+0000 7fb7c4ed3700 1 -- 172.21.15.91:0/931634420 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b0011990 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.741+0000 7fb7c4ed3700 1 -- 172.21.15.91:0/931634420 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0011d20 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.742+0000 7fb7c8139700 1 -- 172.21.15.91:0/931634420 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7c00cf2b0 msgr2=0x7fb7c00d01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.742+0000 7fb7c8139700 1 --2- 172.21.15.91:0/931634420 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7c00cf2b0 0x7fb7c00d01b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0072370 tx=0x7fb7b0009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.742+0000 7fb7c8139700 1 -- 172.21.15.91:0/931634420 shutdown_connections 2023-11-24T06:08:11.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.742+0000 7fb7c8139700 1 --2- 172.21.15.91:0/931634420 >> v2:172.21.15.94:3300/0 conn(0x7fb7c015bde0 0x7fb7c015c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.742+0000 7fb7c8139700 1 --2- 172.21.15.91:0/931634420 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7c00cf2b0 0x7fb7c00d01b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.742+0000 7fb7c8139700 1 -- 172.21.15.91:0/931634420 >> 172.21.15.91:0/931634420 conn(0x7fb7c00b3720 msgr2=0x7fb7c00b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:11.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.742+0000 7fb7c8139700 1 -- 172.21.15.91:0/931634420 shutdown_connections 2023-11-24T06:08:11.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.742+0000 7fb7c8139700 1 -- 172.21.15.91:0/931634420 wait complete. 2023-11-24T06:08:11.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.743+0000 7fb7c8139700 1 Processor -- start 2023-11-24T06:08:11.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.743+0000 7fb7c8139700 1 -- start start 2023-11-24T06:08:11.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c8139700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00ce5c0 0x7fb7c00b4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c8139700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7c00cf2b0 0x7fb7c01f4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c8139700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7c015bde0 0x7fb7c00b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c8139700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c01f5410 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c8139700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c00b22f0 con 0x7fb7c00ce5c0 2023-11-24T06:08:11.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c8139700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c00b2550 con 0x7fb7c015bde0 2023-11-24T06:08:11.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c56d4700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7c00cf2b0 0x7fb7c01f4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c56d4700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7c00cf2b0 0x7fb7c01f4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58254/0 (socket says 172.21.15.91:58254) 2023-11-24T06:08:11.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c56d4700 1 -- 172.21.15.91:0/3394897340 learned_addr learned my addr 172.21.15.91:0/3394897340 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:11.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c5ed5700 1 --2- 172.21.15.91:0/3394897340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00ce5c0 0x7fb7c00b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c56d4700 1 -- 172.21.15.91:0/3394897340 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7c015bde0 msgr2=0x7fb7c00b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c66d6700 1 --2- 172.21.15.91:0/3394897340 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7c015bde0 0x7fb7c00b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c56d4700 1 --2- 172.21.15.91:0/3394897340 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7c015bde0 0x7fb7c00b1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c56d4700 1 -- 172.21.15.91:0/3394897340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00ce5c0 msgr2=0x7fb7c00b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c56d4700 1 --2- 172.21.15.91:0/3394897340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00ce5c0 0x7fb7c00b4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c56d4700 1 -- 172.21.15.91:0/3394897340 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b0009550 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c5ed5700 1 --2- 172.21.15.91:0/3394897340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00ce5c0 0x7fb7c00b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:11.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c66d6700 1 --2- 172.21.15.91:0/3394897340 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7c015bde0 0x7fb7c00b1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:11.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7c56d4700 1 --2- 172.21.15.91:0/3394897340 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7c00cf2b0 0x7fb7c01f4ed0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0000c00 tx=0x7fb7b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:11.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.744+0000 7fb7b67fc700 1 -- 172.21.15.91:0/3394897340 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0004570 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.745+0000 7fb7b67fc700 1 -- 172.21.15.91:0/3394897340 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b001f070 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.745+0000 7fb7c8139700 1 -- 172.21.15.91:0/3394897340 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c00b6310 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.753 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.745+0000 7fb7b67fc700 1 -- 172.21.15.91:0/3394897340 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b00038e0 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.753 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.745+0000 7fb7c8139700 1 -- 172.21.15.91:0/3394897340 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c00b68d0 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.753 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.746+0000 7fb7c8139700 1 -- 172.21.15.91:0/3394897340 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c0075480 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.753 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.746+0000 7fb7b67fc700 1 -- 172.21.15.91:0/3394897340 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb7b0021030 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.754 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.746+0000 7fb7b67fc700 1 --2- 172.21.15.91:0/3394897340 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb7a406aed0 0x7fb7a406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:11.754 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.747+0000 7fb7b67fc700 1 -- 172.21.15.91:0/3394897340 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7b008aae0 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.754 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.747+0000 7fb7c5ed5700 1 --2- 172.21.15.91:0/3394897340 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb7a406aed0 0x7fb7a406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:11.754 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.747+0000 7fb7c5ed5700 1 --2- 172.21.15.91:0/3394897340 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb7a406aed0 0x7fb7a406d390 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009ac0 tx=0x7fb7bc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:11.754 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.751+0000 7fb7b67fc700 1 -- 172.21.15.91:0/3394897340 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb7b0055b80 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.948 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.944+0000 7fb7c8139700 1 -- 172.21.15.91:0/3394897340 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb7c015ac20 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.948 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.944+0000 7fb7b67fc700 1 -- 172.21.15.91:0/3394897340 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb7b0015020 con 0x7fb7c00cf2b0 2023-11-24T06:08:11.948 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:08:11.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.946+0000 7fb7c8139700 1 -- 172.21.15.91:0/3394897340 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb7a406aed0 msgr2=0x7fb7a406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.946+0000 7fb7c8139700 1 --2- 172.21.15.91:0/3394897340 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb7a406aed0 0x7fb7a406d390 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc009ac0 tx=0x7fb7bc008040 comp rx=0 tx=0).stop 2023-11-24T06:08:11.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.946+0000 7fb7c8139700 1 -- 172.21.15.91:0/3394897340 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7c00cf2b0 msgr2=0x7fb7c01f4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:11.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.946+0000 7fb7c8139700 1 --2- 172.21.15.91:0/3394897340 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7c00cf2b0 0x7fb7c01f4ed0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0000c00 tx=0x7fb7b0004750 comp rx=0 tx=0).stop 2023-11-24T06:08:11.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.946+0000 7fb7c8139700 1 -- 172.21.15.91:0/3394897340 shutdown_connections 2023-11-24T06:08:11.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.946+0000 7fb7c8139700 1 --2- 172.21.15.91:0/3394897340 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb7a406aed0 0x7fb7a406d390 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.946+0000 7fb7c8139700 1 --2- 172.21.15.91:0/3394897340 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7c00ce5c0 0x7fb7c00b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.946+0000 7fb7c8139700 1 --2- 172.21.15.91:0/3394897340 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7c015bde0 0x7fb7c00b1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.946+0000 7fb7c8139700 1 --2- 172.21.15.91:0/3394897340 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7c00cf2b0 0x7fb7c01f4ed0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:11.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.946+0000 7fb7c8139700 1 -- 172.21.15.91:0/3394897340 >> 172.21.15.91:0/3394897340 conn(0x7fb7c00b3720 msgr2=0x7fb7c00b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:11.952 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.946+0000 7fb7c8139700 1 -- 172.21.15.91:0/3394897340 shutdown_connections 2023-11-24T06:08:11.952 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:11.946+0000 7fb7c8139700 1 -- 172.21.15.91:0/3394897340 wait complete. 2023-11-24T06:08:11.962 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:08:12.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.081+0000 7fe9d8db9700 1 Processor -- start 2023-11-24T06:08:12.086 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.082+0000 7fe9d8db9700 1 -- start start 2023-11-24T06:08:12.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.082+0000 7fe9d8db9700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe9d415a440 0x7fe9d40cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.082+0000 7fe9d8db9700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d40b05f0 con 0x7fe9d415eb80 2023-11-24T06:08:12.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.082+0000 7fe9d8db9700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d40734c0 con 0x7fe9d415a820 2023-11-24T06:08:12.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.082+0000 7fe9d8db9700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d415f080 con 0x7fe9d415a440 2023-11-24T06:08:12.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.083+0000 7fe9d2d9d700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fe9d415eb80 0x7fe9d4072db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53198/0 (socket says 172.21.15.91:53198) 2023-11-24T06:08:12.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.083+0000 7fe9d2d9d700 1 -- 172.21.15.91:0/4256169784 learned_addr learned my addr 172.21.15.91:0/4256169784 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:12.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.083+0000 7fe9d259c700 1 --2- 172.21.15.91:0/4256169784 >> v2:172.21.15.94:3300/0 conn(0x7fe9d415a440 0x7fe9d40cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.083+0000 7fe9d159a700 1 -- 172.21.15.91:0/4256169784 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308278361 0 0) 0x7fe9d40b05f0 con 0x7fe9d415eb80 2023-11-24T06:08:12.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.083+0000 7fe9d159a700 1 -- 172.21.15.91:0/4256169784 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9bc0035b0 con 0x7fe9d415eb80 2023-11-24T06:08:12.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.083+0000 7fe9d259c700 1 -- 172.21.15.91:0/4256169784 >> v1:172.21.15.94:6789/0 conn(0x7fe9d415a820 legacy=0x7fe9d40726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.083+0000 7fe9d159a700 1 -- 172.21.15.91:0/4256169784 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 663595390 0 0) 0x7fe9bc0035b0 con 0x7fe9d415eb80 2023-11-24T06:08:12.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.083+0000 7fe9d259c700 1 -- 172.21.15.91:0/4256169784 >> v1:172.21.15.91:6789/0 conn(0x7fe9d415eb80 legacy=0x7fe9d4072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.083+0000 7fe9d259c700 1 -- 172.21.15.91:0/4256169784 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d40afd50 con 0x7fe9d415a440 2023-11-24T06:08:12.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.083+0000 7fe9d259c700 1 --2- 172.21.15.91:0/4256169784 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9d415a440 0x7fe9d40cffa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009840 tx=0x7fe9c8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=825201ff22bc61ee server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:12.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.083+0000 7fe9d159a700 1 -- 172.21.15.91:0/4256169784 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc0019f0 con 0x7fe9d415a440 2023-11-24T06:08:12.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.083+0000 7fe9d159a700 1 -- 172.21.15.91:0/4256169784 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c8004030 con 0x7fe9d415a440 2023-11-24T06:08:12.089 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.083+0000 7fe9d159a700 1 -- 172.21.15.91:0/4256169784 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c80043c0 con 0x7fe9d415a440 2023-11-24T06:08:12.092 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.084+0000 7fe9d8db9700 1 -- 172.21.15.91:0/4256169784 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9d415a440 msgr2=0x7fe9d40cffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.092 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.084+0000 7fe9d8db9700 1 --2- 172.21.15.91:0/4256169784 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9d415a440 0x7fe9d40cffa0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8009840 tx=0x7fe9c8009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:12.092 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.084+0000 7fe9d8db9700 1 -- 172.21.15.91:0/4256169784 shutdown_connections 2023-11-24T06:08:12.092 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.084+0000 7fe9d8db9700 1 --2- 172.21.15.91:0/4256169784 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9d415a440 0x7fe9d40cffa0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.092 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.084+0000 7fe9d8db9700 1 -- 172.21.15.91:0/4256169784 >> 172.21.15.91:0/4256169784 conn(0x7fe9d40b5070 msgr2=0x7fe9d40b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:12.092 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.084+0000 7fe9d8db9700 1 -- 172.21.15.91:0/4256169784 shutdown_connections 2023-11-24T06:08:12.092 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.084+0000 7fe9d8db9700 1 -- 172.21.15.91:0/4256169784 wait complete. 2023-11-24T06:08:12.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.085+0000 7fe9d8db9700 1 Processor -- start 2023-11-24T06:08:12.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.085+0000 7fe9d8db9700 1 -- start start 2023-11-24T06:08:12.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.086+0000 7fe9d8db9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9d415a440 0x7fe9d40ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.086+0000 7fe9d8db9700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe9d415eb80 0x7fe9d40999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.086+0000 7fe9d8db9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe9d40cd600 0x7fe9d415ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.086+0000 7fe9d8db9700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d4099ef0 con 0x7fe9d415eb80 2023-11-24T06:08:12.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.086+0000 7fe9d8db9700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d40ae3e0 con 0x7fe9d415a440 2023-11-24T06:08:12.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.086+0000 7fe9d8db9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d40ad7d0 con 0x7fe9d40cd600 2023-11-24T06:08:12.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.086+0000 7fe9d1d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe9d415eb80 0x7fe9d40999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.086+0000 7fe9d1d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe9d415eb80 0x7fe9d40999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58266/0 (socket says 172.21.15.91:58266) 2023-11-24T06:08:12.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.086+0000 7fe9d1d9b700 1 -- 172.21.15.91:0/3483229529 learned_addr learned my addr 172.21.15.91:0/3483229529 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:12.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.086+0000 7fe9d259c700 1 --2- 172.21.15.91:0/3483229529 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9d415a440 0x7fe9d40ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.086+0000 7fe9d1d9b700 1 -- 172.21.15.91:0/3483229529 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe9d40cd600 msgr2=0x7fe9d415ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.087+0000 7fe9d1d9b700 1 --2- 172.21.15.91:0/3483229529 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe9d40cd600 0x7fe9d415ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.087+0000 7fe9d1d9b700 1 -- 172.21.15.91:0/3483229529 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9d415a440 msgr2=0x7fe9d40ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.087+0000 7fe9d1d9b700 1 --2- 172.21.15.91:0/3483229529 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9d415a440 0x7fe9d40ad1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.087+0000 7fe9d1d9b700 1 -- 172.21.15.91:0/3483229529 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c80094a0 con 0x7fe9d415eb80 2023-11-24T06:08:12.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.087+0000 7fe9d1d9b700 1 --2- 172.21.15.91:0/3483229529 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe9d415eb80 0x7fe9d40999b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400c4b0 tx=0x7fe9c400c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:12.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.087+0000 7fe9c2ffd700 1 -- 172.21.15.91:0/3483229529 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c40052d0 con 0x7fe9d415eb80 2023-11-24T06:08:12.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.087+0000 7fe9c2ffd700 1 -- 172.21.15.91:0/3483229529 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c4005430 con 0x7fe9d415eb80 2023-11-24T06:08:12.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.087+0000 7fe9d8db9700 1 -- 172.21.15.91:0/3483229529 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d40adab0 con 0x7fe9d415eb80 2023-11-24T06:08:12.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.087+0000 7fe9c2ffd700 1 -- 172.21.15.91:0/3483229529 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c40038c0 con 0x7fe9d415eb80 2023-11-24T06:08:12.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.087+0000 7fe9d8db9700 1 -- 172.21.15.91:0/3483229529 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d40ae070 con 0x7fe9d415eb80 2023-11-24T06:08:12.095 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.088+0000 7fe9d8db9700 1 -- 172.21.15.91:0/3483229529 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d4075480 con 0x7fe9d415eb80 2023-11-24T06:08:12.095 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.088+0000 7fe9c2ffd700 1 -- 172.21.15.91:0/3483229529 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe9c4003a20 con 0x7fe9d415eb80 2023-11-24T06:08:12.095 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.088+0000 7fe9c2ffd700 1 --2- 172.21.15.91:0/3483229529 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe9b406aed0 0x7fe9b406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.095 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.089+0000 7fe9c2ffd700 1 -- 172.21.15.91:0/3483229529 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9c4088b20 con 0x7fe9d415eb80 2023-11-24T06:08:12.095 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.089+0000 7fe9d259c700 1 --2- 172.21.15.91:0/3483229529 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe9b406aed0 0x7fe9b406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.095 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.089+0000 7fe9d259c700 1 --2- 172.21.15.91:0/3483229529 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe9b406aed0 0x7fe9b406d390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe9c80090b0 tx=0x7fe9c8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:12.096 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.093+0000 7fe9c2ffd700 1 -- 172.21.15.91:0/3483229529 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe9c4053b50 con 0x7fe9d415eb80 2023-11-24T06:08:12.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.282+0000 7fe9d8db9700 1 -- 172.21.15.91:0/3483229529 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe9d40af940 con 0x7fe9d415eb80 2023-11-24T06:08:12.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.282+0000 7fe9c2ffd700 1 -- 172.21.15.91:0/3483229529 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe9c4057170 con 0x7fe9d415eb80 2023-11-24T06:08:12.287 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 32 2023-11-24T06:08:12.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.284+0000 7fe9d8db9700 1 -- 172.21.15.91:0/3483229529 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe9b406aed0 msgr2=0x7fe9b406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.284+0000 7fe9d8db9700 1 --2- 172.21.15.91:0/3483229529 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe9b406aed0 0x7fe9b406d390 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe9c80090b0 tx=0x7fe9c8008030 comp rx=0 tx=0).stop 2023-11-24T06:08:12.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.284+0000 7fe9d8db9700 1 -- 172.21.15.91:0/3483229529 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe9d415eb80 msgr2=0x7fe9d40999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.284+0000 7fe9d8db9700 1 --2- 172.21.15.91:0/3483229529 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe9d415eb80 0x7fe9d40999b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400c4b0 tx=0x7fe9c400c870 comp rx=0 tx=0).stop 2023-11-24T06:08:12.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.284+0000 7fe9d8db9700 1 -- 172.21.15.91:0/3483229529 shutdown_connections 2023-11-24T06:08:12.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.284+0000 7fe9d8db9700 1 --2- 172.21.15.91:0/3483229529 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe9b406aed0 0x7fe9b406d390 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.284+0000 7fe9d8db9700 1 --2- 172.21.15.91:0/3483229529 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe9d415a440 0x7fe9d40ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.284+0000 7fe9d8db9700 1 --2- 172.21.15.91:0/3483229529 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe9d40cd600 0x7fe9d415ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.284+0000 7fe9d8db9700 1 --2- 172.21.15.91:0/3483229529 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe9d415eb80 0x7fe9d40999b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.284+0000 7fe9d8db9700 1 -- 172.21.15.91:0/3483229529 >> 172.21.15.91:0/3483229529 conn(0x7fe9d40b5070 msgr2=0x7fe9d406d350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:12.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.284+0000 7fe9d8db9700 1 -- 172.21.15.91:0/3483229529 shutdown_connections 2023-11-24T06:08:12.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.285+0000 7fe9d8db9700 1 -- 172.21.15.91:0/3483229529 wait complete. 2023-11-24T06:08:12.300 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:08:12.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.419+0000 7fd17b841700 1 Processor -- start 2023-11-24T06:08:12.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.420+0000 7fd17b841700 1 -- start start 2023-11-24T06:08:12.426 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.420+0000 7fd17b841700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd1740722c0 0x7fd1740b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.426 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.420+0000 7fd17b841700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1741440f0 0x7fd17406e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.426 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.420+0000 7fd17b841700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd174143ec0 con 0x7fd174143ae0 2023-11-24T06:08:12.426 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.420+0000 7fd17b841700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1740b40a0 con 0x7fd1740722c0 2023-11-24T06:08:12.426 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.420+0000 7fd17b841700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1740b3ae0 con 0x7fd1741440f0 2023-11-24T06:08:12.426 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.421+0000 7fd178ddc700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd1740722c0 0x7fd1740b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.426 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.421+0000 7fd178ddc700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd1740722c0 0x7fd1740b51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58276/0 (socket says 172.21.15.91:58276) 2023-11-24T06:08:12.427 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.421+0000 7fd178ddc700 1 -- 172.21.15.91:0/1120206590 learned_addr learned my addr 172.21.15.91:0/1120206590 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:12.427 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.421+0000 7fd179dde700 1 --2- 172.21.15.91:0/1120206590 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1741440f0 0x7fd17406e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.427 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.421+0000 7fd178ddc700 1 -- 172.21.15.91:0/1120206590 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1741440f0 msgr2=0x7fd17406e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:12.427 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.421+0000 7fd178ddc700 1 --2- 172.21.15.91:0/1120206590 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1741440f0 0x7fd17406e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.427 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.421+0000 7fd178ddc700 1 -- 172.21.15.91:0/1120206590 >> v1:172.21.15.94:6789/0 conn(0x7fd174143ae0 legacy=0x7fd174071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.427 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.421+0000 7fd178ddc700 1 -- 172.21.15.91:0/1120206590 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1740b3dc0 con 0x7fd1740722c0 2023-11-24T06:08:12.427 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.421+0000 7fd178ddc700 1 --2- 172.21.15.91:0/1120206590 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd1740722c0 0x7fd1740b51b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd1640098f0 tx=0x7fd164009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4844d0421aab9a2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:12.427 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.421+0000 7fd16bfff700 1 -- 172.21.15.91:0/1120206590 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16400b8a0 con 0x7fd1740722c0 2023-11-24T06:08:12.427 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.421+0000 7fd16bfff700 1 -- 172.21.15.91:0/1120206590 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16400ba00 con 0x7fd1740722c0 2023-11-24T06:08:12.427 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.421+0000 7fd16bfff700 1 -- 172.21.15.91:0/1120206590 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16400bd90 con 0x7fd1740722c0 2023-11-24T06:08:12.427 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.422+0000 7fd17b841700 1 -- 172.21.15.91:0/1120206590 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd1740722c0 msgr2=0x7fd1740b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.428 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.422+0000 7fd17b841700 1 --2- 172.21.15.91:0/1120206590 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd1740722c0 0x7fd1740b51b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd1640098f0 tx=0x7fd164009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:12.428 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.422+0000 7fd17b841700 1 -- 172.21.15.91:0/1120206590 shutdown_connections 2023-11-24T06:08:12.428 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.422+0000 7fd17b841700 1 --2- 172.21.15.91:0/1120206590 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1741440f0 0x7fd17406e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.428 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.422+0000 7fd17b841700 1 --2- 172.21.15.91:0/1120206590 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd1740722c0 0x7fd1740b51b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.428 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.422+0000 7fd17b841700 1 -- 172.21.15.91:0/1120206590 >> 172.21.15.91:0/1120206590 conn(0x7fd1740b2740 msgr2=0x7fd1740b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:12.428 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.422+0000 7fd17b841700 1 -- 172.21.15.91:0/1120206590 shutdown_connections 2023-11-24T06:08:12.428 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.422+0000 7fd17b841700 1 -- 172.21.15.91:0/1120206590 wait complete. 2023-11-24T06:08:12.431 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.423+0000 7fd17b841700 1 Processor -- start 2023-11-24T06:08:12.431 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.423+0000 7fd17b841700 1 -- start start 2023-11-24T06:08:12.431 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd17b841700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd1740722c0 0x7fd174153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.431 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd17b841700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd174143ae0 0x7fd174140860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.431 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd17b841700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1741440f0 0x7fd174155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.431 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd17b841700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd174154110 con 0x7fd1740722c0 2023-11-24T06:08:12.432 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd17b841700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741f0ca0 con 0x7fd174143ae0 2023-11-24T06:08:12.432 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd17b841700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741f0f80 con 0x7fd1741440f0 2023-11-24T06:08:12.432 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd1795dd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd1740722c0 0x7fd174153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.432 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd1795dd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd1740722c0 0x7fd174153bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58280/0 (socket says 172.21.15.91:58280) 2023-11-24T06:08:12.432 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd1795dd700 1 -- 172.21.15.91:0/1730649062 learned_addr learned my addr 172.21.15.91:0/1730649062 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:12.432 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd178ddc700 1 --2- 172.21.15.91:0/1730649062 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd174143ae0 0x7fd174140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.432 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd179dde700 1 --2- 172.21.15.91:0/1730649062 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1741440f0 0x7fd174155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.433 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd1795dd700 1 -- 172.21.15.91:0/1730649062 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1741440f0 msgr2=0x7fd174155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.433 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd1795dd700 1 --2- 172.21.15.91:0/1730649062 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1741440f0 0x7fd174155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.433 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd1795dd700 1 -- 172.21.15.91:0/1730649062 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd174143ae0 msgr2=0x7fd174140860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.433 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd1795dd700 1 --2- 172.21.15.91:0/1730649062 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd174143ae0 0x7fd174140860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.433 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd1795dd700 1 -- 172.21.15.91:0/1730649062 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd164009550 con 0x7fd1740722c0 2023-11-24T06:08:12.433 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd1795dd700 1 --2- 172.21.15.91:0/1730649062 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd1740722c0 0x7fd174153bd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd17000c4e0 tx=0x7fd17000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:12.433 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd169ffb700 1 -- 172.21.15.91:0/1730649062 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1700052d0 con 0x7fd1740722c0 2023-11-24T06:08:12.433 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd169ffb700 1 -- 172.21.15.91:0/1730649062 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd170005430 con 0x7fd1740722c0 2023-11-24T06:08:12.433 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.424+0000 7fd17b841700 1 -- 172.21.15.91:0/1730649062 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1741f1260 con 0x7fd1740722c0 2023-11-24T06:08:12.433 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.425+0000 7fd169ffb700 1 -- 172.21.15.91:0/1730649062 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1700038c0 con 0x7fd1740722c0 2023-11-24T06:08:12.433 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.425+0000 7fd17b841700 1 -- 172.21.15.91:0/1730649062 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd174140da0 con 0x7fd1740722c0 2023-11-24T06:08:12.433 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.426+0000 7fd17b841700 1 -- 172.21.15.91:0/1730649062 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1740b37b0 con 0x7fd1740722c0 2023-11-24T06:08:12.434 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.426+0000 7fd169ffb700 1 -- 172.21.15.91:0/1730649062 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd170003a20 con 0x7fd1740722c0 2023-11-24T06:08:12.434 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.426+0000 7fd169ffb700 1 --2- 172.21.15.91:0/1730649062 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd15806aed0 0x7fd15806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.434 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.426+0000 7fd178ddc700 1 --2- 172.21.15.91:0/1730649062 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd15806aed0 0x7fd15806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.434 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.426+0000 7fd169ffb700 1 -- 172.21.15.91:0/1730649062 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1700888d0 con 0x7fd1740722c0 2023-11-24T06:08:12.434 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.427+0000 7fd178ddc700 1 --2- 172.21.15.91:0/1730649062 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd15806aed0 0x7fd15806d390 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd164009c40 tx=0x7fd164008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:12.434 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.431+0000 7fd169ffb700 1 -- 172.21.15.91:0/1730649062 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd170053a10 con 0x7fd1740722c0 2023-11-24T06:08:12.624 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.621+0000 7fd17b841700 1 -- 172.21.15.91:0/1730649062 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd174075480 con 0x7fd1740722c0 2023-11-24T06:08:12.625 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.621+0000 7fd169ffb700 1 -- 172.21.15.91:0/1730649062 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd1700170a0 con 0x7fd1740722c0 2023-11-24T06:08:12.625 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 64 2023-11-24T06:08:12.627 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.623+0000 7fd17b841700 1 -- 172.21.15.91:0/1730649062 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd15806aed0 msgr2=0x7fd15806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.627 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.623+0000 7fd17b841700 1 --2- 172.21.15.91:0/1730649062 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd15806aed0 0x7fd15806d390 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd164009c40 tx=0x7fd164008030 comp rx=0 tx=0).stop 2023-11-24T06:08:12.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.623+0000 7fd17b841700 1 -- 172.21.15.91:0/1730649062 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd1740722c0 msgr2=0x7fd174153bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.623+0000 7fd17b841700 1 --2- 172.21.15.91:0/1730649062 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd1740722c0 0x7fd174153bd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd17000c4e0 tx=0x7fd17000c7f0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.623+0000 7fd17b841700 1 -- 172.21.15.91:0/1730649062 shutdown_connections 2023-11-24T06:08:12.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.623+0000 7fd17b841700 1 --2- 172.21.15.91:0/1730649062 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd15806aed0 0x7fd15806d390 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.623+0000 7fd17b841700 1 --2- 172.21.15.91:0/1730649062 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd1741440f0 0x7fd174155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.623+0000 7fd17b841700 1 --2- 172.21.15.91:0/1730649062 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd1740722c0 0x7fd174153bd0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.623+0000 7fd17b841700 1 --2- 172.21.15.91:0/1730649062 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd174143ae0 0x7fd174140860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.623+0000 7fd17b841700 1 -- 172.21.15.91:0/1730649062 >> 172.21.15.91:0/1730649062 conn(0x7fd1740b2740 msgr2=0x7fd174141840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:12.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.623+0000 7fd17b841700 1 -- 172.21.15.91:0/1730649062 shutdown_connections 2023-11-24T06:08:12.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.623+0000 7fd17b841700 1 -- 172.21.15.91:0/1730649062 wait complete. 2023-11-24T06:08:12.639 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:08:12.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.755+0000 7f18114b5700 1 Processor -- start 2023-11-24T06:08:12.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.755+0000 7f18114b5700 1 -- start start 2023-11-24T06:08:12.764 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.755+0000 7f18114b5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f180c161850 0x7f180c0b5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.765 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.755+0000 7f18114b5700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f180c0af9f0 0x7f180c06afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.765 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.755+0000 7f18114b5700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f180c0b59b0 con 0x7f180c0b29a0 2023-11-24T06:08:12.765 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.755+0000 7f18114b5700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f180c0670c0 con 0x7f180c0af9f0 2023-11-24T06:08:12.765 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.755+0000 7f18114b5700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180c0b5080 con 0x7f180c161850 2023-11-24T06:08:12.765 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f180b7fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f180c0b29a0 0x7f180c0ce640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53226/0 (socket says 172.21.15.91:53226) 2023-11-24T06:08:12.765 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f180b7fe700 1 -- 172.21.15.91:0/1067677373 learned_addr learned my addr 172.21.15.91:0/1067677373 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:12.765 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f180a7fc700 1 --2- 172.21.15.91:0/1067677373 >> v2:172.21.15.94:3300/0 conn(0x7f180c0af9f0 0x7f180c06afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.765 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f180affd700 1 --2- 172.21.15.91:0/1067677373 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f180c161850 0x7f180c0b5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.765 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f1809ffb700 1 -- 172.21.15.91:0/1067677373 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264462536 0 0) 0x7f180c0b59b0 con 0x7f180c0b29a0 2023-11-24T06:08:12.765 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f1809ffb700 1 -- 172.21.15.91:0/1067677373 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17f40035b0 con 0x7f180c0b29a0 2023-11-24T06:08:12.766 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f1809ffb700 1 -- 172.21.15.91:0/1067677373 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1645944097 0 0) 0x7f17f40035b0 con 0x7f180c0b29a0 2023-11-24T06:08:12.766 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f1809ffb700 1 -- 172.21.15.91:0/1067677373 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f180c161850 msgr2=0x7f180c0b5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:12.766 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f1809ffb700 1 --2- 172.21.15.91:0/1067677373 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f180c161850 0x7f180c0b5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.766 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f1809ffb700 1 -- 172.21.15.91:0/1067677373 >> v2:172.21.15.94:3300/0 conn(0x7f180c0af9f0 msgr2=0x7f180c06afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:12.766 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f1809ffb700 1 --2- 172.21.15.91:0/1067677373 >> v2:172.21.15.94:3300/0 conn(0x7f180c0af9f0 0x7f180c06afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.766 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f1809ffb700 1 -- 172.21.15.91:0/1067677373 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180c0b5360 con 0x7f180c0b29a0 2023-11-24T06:08:12.766 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f1809ffb700 1 -- 172.21.15.91:0/1067677373 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f18040019f0 con 0x7f180c0b29a0 2023-11-24T06:08:12.766 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f180affd700 1 --2- 172.21.15.91:0/1067677373 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f180c161850 0x7f180c0b5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:08:12.766 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f180a7fc700 1 --2- 172.21.15.91:0/1067677373 >> v2:172.21.15.94:3300/0 conn(0x7f180c0af9f0 0x7f180c06afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:08:12.766 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f1809ffb700 1 -- 172.21.15.91:0/1067677373 >> v1:172.21.15.91:6789/0 conn(0x7f180c0b29a0 legacy=0x7f180c0ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.767 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f1809ffb700 1 --2- 172.21.15.91:0/1067677373 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f180c161850 0x7f17f4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.767 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f1809ffb700 1 --2- 172.21.15.91:0/1067677373 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f17f4003f40 0x7f17f4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.767 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f1809ffb700 1 --2- 172.21.15.91:0/1067677373 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17f4004d90 0x7f17f4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.767 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f1809ffb700 1 -- 172.21.15.91:0/1067677373 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180c0670c0 con 0x7f17f4003f40 2023-11-24T06:08:12.767 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f1809ffb700 1 -- 172.21.15.91:0/1067677373 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180c0b5080 con 0x7f17f4004d90 2023-11-24T06:08:12.767 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f1809ffb700 1 -- 172.21.15.91:0/1067677373 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f400ba10 con 0x7f180c161850 2023-11-24T06:08:12.767 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f180a7fc700 1 --2- 172.21.15.91:0/1067677373 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f17f4003f40 0x7f17f4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.767 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f180affd700 1 --2- 172.21.15.91:0/1067677373 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f180c161850 0x7f17f4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.767 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f180b7fe700 1 --2- 172.21.15.91:0/1067677373 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17f4004d90 0x7f17f4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.767 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f180a7fc700 1 -- 172.21.15.91:0/1067677373 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f180c161850 msgr2=0x7f17f4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:12.767 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.756+0000 7f180a7fc700 1 --2- 172.21.15.91:0/1067677373 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f180c161850 0x7f17f4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.768 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.757+0000 7f180a7fc700 1 -- 172.21.15.91:0/1067677373 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17f4004d90 msgr2=0x7f17f4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:12.768 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.757+0000 7f180a7fc700 1 --2- 172.21.15.91:0/1067677373 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f17f4004d90 0x7f17f4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.768 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.757+0000 7f180a7fc700 1 -- 172.21.15.91:0/1067677373 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17fc0094a0 con 0x7f17f4003f40 2023-11-24T06:08:12.768 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.757+0000 7f180a7fc700 1 --2- 172.21.15.91:0/1067677373 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f17f4003f40 0x7f17f4004850 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1800009d30 tx=0x7f17f4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca9c3da284edf7f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:12.768 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.757+0000 7f1809ffb700 1 -- 172.21.15.91:0/1067677373 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800009e80 con 0x7f17f4003f40 2023-11-24T06:08:12.768 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.757+0000 7f1809ffb700 1 -- 172.21.15.91:0/1067677373 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180000e9a0 con 0x7f17f4003f40 2023-11-24T06:08:12.768 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.757+0000 7f1809ffb700 1 -- 172.21.15.91:0/1067677373 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180000ed30 con 0x7f17f4003f40 2023-11-24T06:08:12.768 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.757+0000 7f18114b5700 1 -- 172.21.15.91:0/1067677373 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f17f4003f40 msgr2=0x7f17f4004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.768 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.757+0000 7f18114b5700 1 --2- 172.21.15.91:0/1067677373 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f17f4003f40 0x7f17f4004850 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1800009d30 tx=0x7f17f4005880 comp rx=0 tx=0).stop 2023-11-24T06:08:12.768 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.757+0000 7f180b7fe700 1 -- 172.21.15.91:0/1067677373 reap_dead start 2023-11-24T06:08:12.768 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.757+0000 7f18114b5700 1 -- 172.21.15.91:0/1067677373 shutdown_connections 2023-11-24T06:08:12.768 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.757+0000 7f18114b5700 1 -- 172.21.15.91:0/1067677373 >> 172.21.15.91:0/1067677373 conn(0x7f180c0b1e80 msgr2=0x7f180c0b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:12.769 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.757+0000 7f18114b5700 1 -- 172.21.15.91:0/1067677373 shutdown_connections 2023-11-24T06:08:12.769 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.757+0000 7f18114b5700 1 -- 172.21.15.91:0/1067677373 wait complete. 2023-11-24T06:08:12.769 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.759+0000 7f18114b5700 1 Processor -- start 2023-11-24T06:08:12.769 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.759+0000 7f18114b5700 1 -- start start 2023-11-24T06:08:12.769 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.759+0000 7f18114b5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f17f4003f40 0x7f180c0c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.769 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.759+0000 7f18114b5700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f180c0ca700 0x7f180c154430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.769 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.759+0000 7f18114b5700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f180c1608c0 0x7f180c160d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.769 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.759+0000 7f18114b5700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180c0cd8d0 con 0x7f180c0ca700 2023-11-24T06:08:12.769 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.759+0000 7f18114b5700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180c1612b0 con 0x7f180c1608c0 2023-11-24T06:08:12.769 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.759+0000 7f18114b5700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f180c0ca420 con 0x7f17f4003f40 2023-11-24T06:08:12.769 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.760+0000 7f180a7fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f180c0ca700 0x7f180c154430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.769 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.760+0000 7f180a7fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f180c0ca700 0x7f180c154430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58302/0 (socket says 172.21.15.91:58302) 2023-11-24T06:08:12.769 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.760+0000 7f180a7fc700 1 -- 172.21.15.91:0/3967555839 learned_addr learned my addr 172.21.15.91:0/3967555839 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:12.770 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.760+0000 7f180b7fe700 1 --2- 172.21.15.91:0/3967555839 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f180c1608c0 0x7f180c160d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.770 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.760+0000 7f180affd700 1 --2- 172.21.15.91:0/3967555839 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f17f4003f40 0x7f180c0c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.770 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.760+0000 7f180a7fc700 1 -- 172.21.15.91:0/3967555839 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f17f4003f40 msgr2=0x7f180c0c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.770 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.760+0000 7f180a7fc700 1 --2- 172.21.15.91:0/3967555839 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f17f4003f40 0x7f180c0c9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.770 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.760+0000 7f180a7fc700 1 -- 172.21.15.91:0/3967555839 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f180c1608c0 msgr2=0x7f180c160d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:12.770 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.760+0000 7f180a7fc700 1 --2- 172.21.15.91:0/3967555839 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f180c1608c0 0x7f180c160d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:12.770 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.760+0000 7f180a7fc700 1 -- 172.21.15.91:0/3967555839 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f180c0c75b0 con 0x7f180c0ca700 2023-11-24T06:08:12.770 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.760+0000 7f180a7fc700 1 --2- 172.21.15.91:0/3967555839 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f180c0ca700 0x7f180c154430 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1800000c00 tx=0x7f18000047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:12.770 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.760+0000 7f17f37fe700 1 -- 172.21.15.91:0/3967555839 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800003ab0 con 0x7f180c0ca700 2023-11-24T06:08:12.770 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.760+0000 7f17f37fe700 1 -- 172.21.15.91:0/3967555839 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1800003c10 con 0x7f180c0ca700 2023-11-24T06:08:12.770 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.760+0000 7f18114b5700 1 -- 172.21.15.91:0/3967555839 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f180c0cd310 con 0x7f180c0ca700 2023-11-24T06:08:12.771 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.760+0000 7f17f37fe700 1 -- 172.21.15.91:0/3967555839 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800009e80 con 0x7f180c0ca700 2023-11-24T06:08:12.771 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.760+0000 7f18114b5700 1 -- 172.21.15.91:0/3967555839 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f180c0cab80 con 0x7f180c0ca700 2023-11-24T06:08:12.772 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.761+0000 7f18114b5700 1 -- 172.21.15.91:0/3967555839 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f180c0c7380 con 0x7f180c0ca700 2023-11-24T06:08:12.772 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.762+0000 7f17f37fe700 1 -- 172.21.15.91:0/3967555839 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1800013470 con 0x7f180c0ca700 2023-11-24T06:08:12.772 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.762+0000 7f17f37fe700 1 --2- 172.21.15.91:0/3967555839 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f17f806aed0 0x7f17f806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:12.772 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.762+0000 7f17f37fe700 1 -- 172.21.15.91:0/3967555839 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f180008b160 con 0x7f180c0ca700 2023-11-24T06:08:12.772 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.762+0000 7f180affd700 1 --2- 172.21.15.91:0/3967555839 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f17f806aed0 0x7f17f806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:12.772 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.763+0000 7f180affd700 1 --2- 172.21.15.91:0/3967555839 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f17f806aed0 0x7f17f806d390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f17fc009b10 tx=0x7f17fc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:12.772 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:12.766+0000 7f17f37fe700 1 -- 172.21.15.91:0/3967555839 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1800059e10 con 0x7f180c0ca700 2023-11-24T06:08:13.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.000+0000 7f18114b5700 1 -- 172.21.15.91:0/3967555839 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f180c067f60 con 0x7f180c0ca700 2023-11-24T06:08:13.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.001+0000 7f17f37fe700 1 -- 172.21.15.91:0/3967555839 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4907 (secure 0 0 0) 0x7f18000599a0 con 0x7f180c0ca700 2023-11-24T06:08:13.005 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:13.005 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1329219967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1329219967},{"type":"v1","addr":"172.21.15.94:6834","nonce":1329219967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3439524864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3439524864},{"type":"v1","addr":"172.21.15.94:6835","nonce":3439524864}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6836/3390368219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6834","nonce":3390368219},{"type":"v1","addr":"172.21.15.91:6836","nonce":3390368219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:08:00.798681+0000","modified":"2023-11-24T06:08:04.132764+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.91:6837/1644079645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":1644079645},{"type":"v1","addr":"172.21.15.91:6837","nonce":1644079645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T06:08:13.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.003+0000 7f18114b5700 1 -- 172.21.15.91:0/3967555839 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f17f806aed0 msgr2=0x7f17f806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.003+0000 7f18114b5700 1 --2- 172.21.15.91:0/3967555839 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f17f806aed0 0x7f17f806d390 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f17fc009b10 tx=0x7f17fc008040 comp rx=0 tx=0).stop 2023-11-24T06:08:13.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.003+0000 7f18114b5700 1 -- 172.21.15.91:0/3967555839 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f180c0ca700 msgr2=0x7f180c154430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.003+0000 7f18114b5700 1 --2- 172.21.15.91:0/3967555839 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f180c0ca700 0x7f180c154430 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1800000c00 tx=0x7f18000047c0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.003+0000 7f18114b5700 1 -- 172.21.15.91:0/3967555839 shutdown_connections 2023-11-24T06:08:13.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.003+0000 7f18114b5700 1 --2- 172.21.15.91:0/3967555839 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f17f806aed0 0x7f17f806d390 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.003+0000 7f18114b5700 1 --2- 172.21.15.91:0/3967555839 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f180c1608c0 0x7f180c160d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.003+0000 7f18114b5700 1 --2- 172.21.15.91:0/3967555839 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f17f4003f40 0x7f180c0c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.003+0000 7f18114b5700 1 --2- 172.21.15.91:0/3967555839 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f180c0ca700 0x7f180c154430 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.003+0000 7f18114b5700 1 -- 172.21.15.91:0/3967555839 >> 172.21.15.91:0/3967555839 conn(0x7f180c0b1e80 msgr2=0x7f180c06e770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:13.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.003+0000 7f18114b5700 1 -- 172.21.15.91:0/3967555839 shutdown_connections 2023-11-24T06:08:13.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.003+0000 7f18114b5700 1 -- 172.21.15.91:0/3967555839 wait complete. 2023-11-24T06:08:13.008 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 8 2023-11-24T06:08:13.019 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:13.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.135+0000 7f730f1fa700 1 Processor -- start 2023-11-24T06:08:13.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.135+0000 7f730f1fa700 1 -- start start 2023-11-24T06:08:13.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.135+0000 7f730f1fa700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f730806fc50 0x7f730806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.135+0000 7f730f1fa700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f73080b0e90 0x7f730806ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.135+0000 7f730f1fa700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73080d1230 con 0x7f730815e1c0 2023-11-24T06:08:13.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f730f1fa700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73080d0c70 con 0x7f73080b0e90 2023-11-24T06:08:13.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f730f1fa700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73080d0f50 con 0x7f730806fc50 2023-11-24T06:08:13.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f730cf96700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f730815e1c0 0x7f730806f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53242/0 (socket says 172.21.15.91:53242) 2023-11-24T06:08:13.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f730cf96700 1 -- 172.21.15.91:0/474504960 learned_addr learned my addr 172.21.15.91:0/474504960 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:13.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f73077fe700 1 -- 172.21.15.91:0/474504960 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761588091 0 0) 0x7f73080d1230 con 0x7f730815e1c0 2023-11-24T06:08:13.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f730d797700 1 --2- 172.21.15.91:0/474504960 >> v2:172.21.15.94:3300/0 conn(0x7f73080b0e90 0x7f730806ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f73077fe700 1 -- 172.21.15.91:0/474504960 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72f80035b0 con 0x7f730815e1c0 2023-11-24T06:08:13.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f7307fff700 1 --2- 172.21.15.91:0/474504960 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f730806fc50 0x7f730806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f73077fe700 1 -- 172.21.15.91:0/474504960 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2668692627 0 0) 0x7f72f80035b0 con 0x7f730815e1c0 2023-11-24T06:08:13.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f73077fe700 1 -- 172.21.15.91:0/474504960 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f730806fc50 msgr2=0x7f730806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:13.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f73077fe700 1 --2- 172.21.15.91:0/474504960 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f730806fc50 0x7f730806eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f73077fe700 1 -- 172.21.15.91:0/474504960 >> v2:172.21.15.94:3300/0 conn(0x7f73080b0e90 msgr2=0x7f730806ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:13.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f73077fe700 1 --2- 172.21.15.91:0/474504960 >> v2:172.21.15.94:3300/0 conn(0x7f73080b0e90 0x7f730806ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f73077fe700 1 -- 172.21.15.91:0/474504960 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730806c210 con 0x7f730815e1c0 2023-11-24T06:08:13.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f73077fe700 1 -- 172.21.15.91:0/474504960 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f72fc001670 con 0x7f730815e1c0 2023-11-24T06:08:13.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f73077fe700 1 -- 172.21.15.91:0/474504960 >> v1:172.21.15.91:6789/0 conn(0x7f730815e1c0 legacy=0x7f730806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f73077fe700 1 --2- 172.21.15.91:0/474504960 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72f8003a00 0x7f72f8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f73077fe700 1 --2- 172.21.15.91:0/474504960 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f730806fc50 0x7f72f8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f73077fe700 1 --2- 172.21.15.91:0/474504960 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72f8004d90 0x7f72f8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f73077fe700 1 -- 172.21.15.91:0/474504960 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73080d0c70 con 0x7f72f8004d90 2023-11-24T06:08:13.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f73077fe700 1 -- 172.21.15.91:0/474504960 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73080d0f50 con 0x7f72f8003a00 2023-11-24T06:08:13.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f73077fe700 1 -- 172.21.15.91:0/474504960 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f800bb70 con 0x7f730806fc50 2023-11-24T06:08:13.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f730d797700 1 --2- 172.21.15.91:0/474504960 >> v2:172.21.15.94:3300/0 conn(0x7f73080b0e90 0x7f730806ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:13.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f730cf96700 1 --2- 172.21.15.91:0/474504960 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72f8003a00 0x7f72f8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f730d797700 1 --2- 172.21.15.91:0/474504960 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72f8004d90 0x7f72f8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.136+0000 7f7307fff700 1 --2- 172.21.15.91:0/474504960 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f730806fc50 0x7f72f8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.137+0000 7f730d797700 1 -- 172.21.15.91:0/474504960 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f730806fc50 msgr2=0x7f72f8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:13.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.137+0000 7f730d797700 1 --2- 172.21.15.91:0/474504960 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f730806fc50 0x7f72f8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.137+0000 7f730d797700 1 -- 172.21.15.91:0/474504960 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72f8003a00 msgr2=0x7f72f8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:13.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.137+0000 7f730d797700 1 --2- 172.21.15.91:0/474504960 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72f8003a00 0x7f72f8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.137+0000 7f730d797700 1 -- 172.21.15.91:0/474504960 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f80035b0 con 0x7f72f8004d90 2023-11-24T06:08:13.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.137+0000 7f730d797700 1 --2- 172.21.15.91:0/474504960 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72f8004d90 0x7f72f8005210 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f73000073b0 tx=0x7f72f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1479fe53d868d6f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:13.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.137+0000 7f73077fe700 1 -- 172.21.15.91:0/474504960 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7300014400 con 0x7f72f8004d90 2023-11-24T06:08:13.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.137+0000 7f73077fe700 1 -- 172.21.15.91:0/474504960 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7300014560 con 0x7f72f8004d90 2023-11-24T06:08:13.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.137+0000 7f73077fe700 1 -- 172.21.15.91:0/474504960 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73000148d0 con 0x7f72f8004d90 2023-11-24T06:08:13.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.137+0000 7f730f1fa700 1 -- 172.21.15.91:0/474504960 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72f8004d90 msgr2=0x7f72f8005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.137+0000 7f730f1fa700 1 --2- 172.21.15.91:0/474504960 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72f8004d90 0x7f72f8005210 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f73000073b0 tx=0x7f72f8005880 comp rx=0 tx=0).stop 2023-11-24T06:08:13.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.137+0000 7f730d797700 1 -- 172.21.15.91:0/474504960 reap_dead start 2023-11-24T06:08:13.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.137+0000 7f730f1fa700 1 -- 172.21.15.91:0/474504960 shutdown_connections 2023-11-24T06:08:13.147 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.137+0000 7f730f1fa700 1 -- 172.21.15.91:0/474504960 >> 172.21.15.91:0/474504960 conn(0x7f73080b3720 msgr2=0x7f73080b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:13.147 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.137+0000 7f730f1fa700 1 -- 172.21.15.91:0/474504960 shutdown_connections 2023-11-24T06:08:13.147 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.137+0000 7f730f1fa700 1 -- 172.21.15.91:0/474504960 wait complete. 2023-11-24T06:08:13.147 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.139+0000 7f730f1fa700 1 Processor -- start 2023-11-24T06:08:13.147 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.139+0000 7f730f1fa700 1 -- start start 2023-11-24T06:08:13.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.139+0000 7f730f1fa700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f730814c0a0 0x7f7308156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.139+0000 7f730f1fa700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73080b47f0 0x7f73080b5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f730f1fa700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73080b6bd0 0x7f73080b7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f730f1fa700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73080b7590 con 0x7f730814c0a0 2023-11-24T06:08:13.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f730f1fa700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73080b5970 con 0x7f73080b47f0 2023-11-24T06:08:13.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f730f1fa700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73080b4d60 con 0x7f73080b6bd0 2023-11-24T06:08:13.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f730cf96700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f730814c0a0 0x7f7308156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f730cf96700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f730814c0a0 0x7f7308156150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58322/0 (socket says 172.21.15.91:58322) 2023-11-24T06:08:13.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f730cf96700 1 -- 172.21.15.91:0/1314340285 learned_addr learned my addr 172.21.15.91:0/1314340285 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:13.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f730cf96700 1 -- 172.21.15.91:0/1314340285 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73080b6bd0 msgr2=0x7f73080b7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f730cf96700 1 --2- 172.21.15.91:0/1314340285 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73080b6bd0 0x7f73080b7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f730cf96700 1 -- 172.21.15.91:0/1314340285 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73080b47f0 msgr2=0x7f73080b5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f7307fff700 1 --2- 172.21.15.91:0/1314340285 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73080b47f0 0x7f73080b5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f730cf96700 1 --2- 172.21.15.91:0/1314340285 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73080b47f0 0x7f73080b5e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f730cf96700 1 -- 172.21.15.91:0/1314340285 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73080b5040 con 0x7f730814c0a0 2023-11-24T06:08:13.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f730cf96700 1 --2- 172.21.15.91:0/1314340285 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f730814c0a0 0x7f7308156150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f72fc001e70 tx=0x7f72fc007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:13.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f73057fa700 1 -- 172.21.15.91:0/1314340285 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc00f8a0 con 0x7f730814c0a0 2023-11-24T06:08:13.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f73057fa700 1 -- 172.21.15.91:0/1314340285 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc00fa20 con 0x7f730814c0a0 2023-11-24T06:08:13.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f73057fa700 1 -- 172.21.15.91:0/1314340285 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc00fdd0 con 0x7f730814c0a0 2023-11-24T06:08:13.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f730f1fa700 1 -- 172.21.15.91:0/1314340285 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73080b5320 con 0x7f730814c0a0 2023-11-24T06:08:13.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.140+0000 7f730f1fa700 1 -- 172.21.15.91:0/1314340285 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73080b6340 con 0x7f730814c0a0 2023-11-24T06:08:13.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.141+0000 7f730f1fa700 1 -- 172.21.15.91:0/1314340285 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7308068670 con 0x7f730814c0a0 2023-11-24T06:08:13.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.141+0000 7f73057fa700 1 -- 172.21.15.91:0/1314340285 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f72fc01d460 con 0x7f730814c0a0 2023-11-24T06:08:13.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.142+0000 7f73057fa700 1 --2- 172.21.15.91:0/1314340285 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f72ec06aed0 0x7f72ec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.142+0000 7f7307fff700 1 --2- 172.21.15.91:0/1314340285 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f72ec06aed0 0x7f72ec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.142+0000 7f73057fa700 1 -- 172.21.15.91:0/1314340285 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72fc089600 con 0x7f730814c0a0 2023-11-24T06:08:13.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.142+0000 7f7307fff700 1 --2- 172.21.15.91:0/1314340285 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f72ec06aed0 0x7f72ec06d390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f73080b2910 tx=0x7f72f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:13.152 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.146+0000 7f73057fa700 1 -- 172.21.15.91:0/1314340285 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f72fc054740 con 0x7f730814c0a0 2023-11-24T06:08:13.342 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.338+0000 7f730f1fa700 1 -- 172.21.15.91:0/1314340285 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73080ced90 con 0x7f730814c0a0 2023-11-24T06:08:13.343 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.340+0000 7f73057fa700 1 -- 172.21.15.91:0/1314340285 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7f72fc057d60 con 0x7f730814c0a0 2023-11-24T06:08:13.344 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:13.344 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":20,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:02.415612+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:13.346 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.342+0000 7f730f1fa700 1 -- 172.21.15.91:0/1314340285 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f72ec06aed0 msgr2=0x7f72ec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.346 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.342+0000 7f730f1fa700 1 --2- 172.21.15.91:0/1314340285 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f72ec06aed0 0x7f72ec06d390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f73080b2910 tx=0x7f72f4008040 comp rx=0 tx=0).stop 2023-11-24T06:08:13.346 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.342+0000 7f730f1fa700 1 -- 172.21.15.91:0/1314340285 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f730814c0a0 msgr2=0x7f7308156150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.346 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.342+0000 7f730f1fa700 1 --2- 172.21.15.91:0/1314340285 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f730814c0a0 0x7f7308156150 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f72fc001e70 tx=0x7f72fc007620 comp rx=0 tx=0).stop 2023-11-24T06:08:13.346 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.342+0000 7f730f1fa700 1 -- 172.21.15.91:0/1314340285 shutdown_connections 2023-11-24T06:08:13.347 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.342+0000 7f730f1fa700 1 --2- 172.21.15.91:0/1314340285 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f72ec06aed0 0x7f72ec06d390 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.347 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.342+0000 7f730f1fa700 1 --2- 172.21.15.91:0/1314340285 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73080b6bd0 0x7f73080b7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.347 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.342+0000 7f730f1fa700 1 --2- 172.21.15.91:0/1314340285 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f730814c0a0 0x7f7308156150 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.347 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.342+0000 7f730f1fa700 1 --2- 172.21.15.91:0/1314340285 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f73080b47f0 0x7f73080b5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.347 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.342+0000 7f730f1fa700 1 -- 172.21.15.91:0/1314340285 >> 172.21.15.91:0/1314340285 conn(0x7f73080b3720 msgr2=0x7f73080b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:13.347 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.342+0000 7f730f1fa700 1 -- 172.21.15.91:0/1314340285 shutdown_connections 2023-11-24T06:08:13.347 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.342+0000 7f730f1fa700 1 -- 172.21.15.91:0/1314340285 wait complete. 2023-11-24T06:08:13.358 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:07:58.710106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:07:59.774819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:08:13.358 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:08:13.482 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.478+0000 7f3a7b115700 1 Processor -- start 2023-11-24T06:08:13.482 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.478+0000 7f3a7b115700 1 -- start start 2023-11-24T06:08:13.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.478+0000 7f3a7b115700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3a7415bde0 0x7f3a7415c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a7b115700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a740cf2b0 0x7f3a740d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a7b115700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a740d0a90 con 0x7f3a740ce5c0 2023-11-24T06:08:13.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a7b115700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a7406ee10 con 0x7f3a7415bde0 2023-11-24T06:08:13.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a7b115700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7406f0f0 con 0x7f3a740cf2b0 2023-11-24T06:08:13.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a796b2700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f3a740ce5c0 0x7f3a740cc670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53266/0 (socket says 172.21.15.91:53266) 2023-11-24T06:08:13.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a796b2700 1 -- 172.21.15.91:0/477922278 learned_addr learned my addr 172.21.15.91:0/477922278 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:13.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a78eb1700 1 --2- 172.21.15.91:0/477922278 >> v2:172.21.15.94:3300/0 conn(0x7f3a7415bde0 0x7f3a7415c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a73fff700 1 --2- 172.21.15.91:0/477922278 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a740cf2b0 0x7f3a740d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a737fe700 1 -- 172.21.15.91:0/477922278 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431536889 0 0) 0x7f3a740d0a90 con 0x7f3a740ce5c0 2023-11-24T06:08:13.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a737fe700 1 -- 172.21.15.91:0/477922278 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a640035b0 con 0x7f3a740ce5c0 2023-11-24T06:08:13.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a737fe700 1 -- 172.21.15.91:0/477922278 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3633509483 0 0) 0x7f3a640035b0 con 0x7f3a740ce5c0 2023-11-24T06:08:13.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a737fe700 1 -- 172.21.15.91:0/477922278 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a740cf2b0 msgr2=0x7f3a740d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:13.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a737fe700 1 --2- 172.21.15.91:0/477922278 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a740cf2b0 0x7f3a740d01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a737fe700 1 -- 172.21.15.91:0/477922278 >> v2:172.21.15.94:3300/0 conn(0x7f3a7415bde0 msgr2=0x7f3a7415c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:13.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a737fe700 1 --2- 172.21.15.91:0/477922278 >> v2:172.21.15.94:3300/0 conn(0x7f3a7415bde0 0x7f3a7415c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a737fe700 1 -- 172.21.15.91:0/477922278 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7406f3d0 con 0x7f3a740ce5c0 2023-11-24T06:08:13.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a737fe700 1 -- 172.21.15.91:0/477922278 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f3a6c0019f0 con 0x7f3a740ce5c0 2023-11-24T06:08:13.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a737fe700 1 -- 172.21.15.91:0/477922278 >> v1:172.21.15.91:6789/0 conn(0x7f3a740ce5c0 legacy=0x7f3a740cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a737fe700 1 --2- 172.21.15.91:0/477922278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a64003a00 0x7f3a64003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a737fe700 1 --2- 172.21.15.91:0/477922278 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a740cf2b0 0x7f3a64004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a737fe700 1 --2- 172.21.15.91:0/477922278 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3a64004d90 0x7f3a64005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a737fe700 1 -- 172.21.15.91:0/477922278 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7406ee10 con 0x7f3a64004d90 2023-11-24T06:08:13.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a737fe700 1 -- 172.21.15.91:0/477922278 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7406f0f0 con 0x7f3a64003a00 2023-11-24T06:08:13.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a737fe700 1 -- 172.21.15.91:0/477922278 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6400ba10 con 0x7f3a740cf2b0 2023-11-24T06:08:13.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a796b2700 1 --2- 172.21.15.91:0/477922278 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3a64004d90 0x7f3a64005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a78eb1700 1 --2- 172.21.15.91:0/477922278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a64003a00 0x7f3a64003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.479+0000 7f3a73fff700 1 --2- 172.21.15.91:0/477922278 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a740cf2b0 0x7f3a64004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.480+0000 7f3a796b2700 1 -- 172.21.15.91:0/477922278 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a740cf2b0 msgr2=0x7f3a64004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:13.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.480+0000 7f3a796b2700 1 --2- 172.21.15.91:0/477922278 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a740cf2b0 0x7f3a64004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.480+0000 7f3a796b2700 1 -- 172.21.15.91:0/477922278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a64003a00 msgr2=0x7f3a64003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:13.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.480+0000 7f3a796b2700 1 --2- 172.21.15.91:0/477922278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a64003a00 0x7f3a64003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.480+0000 7f3a796b2700 1 -- 172.21.15.91:0/477922278 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a640035b0 con 0x7f3a64004d90 2023-11-24T06:08:13.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.480+0000 7f3a796b2700 1 --2- 172.21.15.91:0/477922278 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3a64004d90 0x7f3a64005210 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00a8d0 tx=0x7f3a64005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4c8b828f857f78 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:13.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.480+0000 7f3a737fe700 1 -- 172.21.15.91:0/477922278 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c00a340 con 0x7f3a64004d90 2023-11-24T06:08:13.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.480+0000 7f3a737fe700 1 -- 172.21.15.91:0/477922278 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6c012600 con 0x7f3a64004d90 2023-11-24T06:08:13.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.480+0000 7f3a737fe700 1 -- 172.21.15.91:0/477922278 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c0129b0 con 0x7f3a64004d90 2023-11-24T06:08:13.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.480+0000 7f3a7b115700 1 -- 172.21.15.91:0/477922278 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3a64004d90 msgr2=0x7f3a64005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.480+0000 7f3a7b115700 1 --2- 172.21.15.91:0/477922278 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3a64004d90 0x7f3a64005210 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00a8d0 tx=0x7f3a64005880 comp rx=0 tx=0).stop 2023-11-24T06:08:13.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.480+0000 7f3a796b2700 1 -- 172.21.15.91:0/477922278 reap_dead start 2023-11-24T06:08:13.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.480+0000 7f3a7b115700 1 -- 172.21.15.91:0/477922278 shutdown_connections 2023-11-24T06:08:13.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.480+0000 7f3a7b115700 1 -- 172.21.15.91:0/477922278 >> 172.21.15.91:0/477922278 conn(0x7f3a740b3720 msgr2=0x7f3a740b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:13.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.480+0000 7f3a7b115700 1 -- 172.21.15.91:0/477922278 shutdown_connections 2023-11-24T06:08:13.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.480+0000 7f3a7b115700 1 -- 172.21.15.91:0/477922278 wait complete. 2023-11-24T06:08:13.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.482+0000 7f3a7b115700 1 Processor -- start 2023-11-24T06:08:13.490 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.482+0000 7f3a7b115700 1 -- start start 2023-11-24T06:08:13.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.482+0000 7f3a7b115700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3a741f4bf0 0x7f3a7415e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.482+0000 7f3a7b115700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a7414c2e0 0x7f3a740c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a7b115700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a7415e9b0 0x7f3a740c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a7b115700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a740cb880 con 0x7f3a741f4bf0 2023-11-24T06:08:13.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a7b115700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a740cacb0 con 0x7f3a7415e9b0 2023-11-24T06:08:13.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a7b115700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a740caf50 con 0x7f3a7414c2e0 2023-11-24T06:08:13.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a78eb1700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3a741f4bf0 0x7f3a7415e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a78eb1700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3a741f4bf0 0x7f3a7415e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58340/0 (socket says 172.21.15.91:58340) 2023-11-24T06:08:13.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a78eb1700 1 -- 172.21.15.91:0/684493894 learned_addr learned my addr 172.21.15.91:0/684493894 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:13.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a796b2700 1 --2- 172.21.15.91:0/684493894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a7415e9b0 0x7f3a740c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a73fff700 1 --2- 172.21.15.91:0/684493894 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a7414c2e0 0x7f3a740c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a78eb1700 1 -- 172.21.15.91:0/684493894 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a7414c2e0 msgr2=0x7f3a740c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a78eb1700 1 --2- 172.21.15.91:0/684493894 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a7414c2e0 0x7f3a740c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a78eb1700 1 -- 172.21.15.91:0/684493894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a7415e9b0 msgr2=0x7f3a740c91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a78eb1700 1 --2- 172.21.15.91:0/684493894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a7415e9b0 0x7f3a740c91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a78eb1700 1 -- 172.21.15.91:0/684493894 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a740cb230 con 0x7f3a741f4bf0 2023-11-24T06:08:13.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a796b2700 1 --2- 172.21.15.91:0/684493894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a7415e9b0 0x7f3a740c91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:13.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a78eb1700 1 --2- 172.21.15.91:0/684493894 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3a741f4bf0 0x7f3a7415e470 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3a68011aa0 tx=0x7f3a68011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:13.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a717fa700 1 -- 172.21.15.91:0/684493894 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6800f980 con 0x7f3a741f4bf0 2023-11-24T06:08:13.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a717fa700 1 -- 172.21.15.91:0/684493894 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a6800fb00 con 0x7f3a741f4bf0 2023-11-24T06:08:13.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a7b115700 1 -- 172.21.15.91:0/684493894 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a740cb510 con 0x7f3a741f4bf0 2023-11-24T06:08:13.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a717fa700 1 -- 172.21.15.91:0/684493894 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6800feb0 con 0x7f3a741f4bf0 2023-11-24T06:08:13.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.483+0000 7f3a7b115700 1 -- 172.21.15.91:0/684493894 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a741f4fd0 con 0x7f3a741f4bf0 2023-11-24T06:08:13.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.484+0000 7f3a7b115700 1 -- 172.21.15.91:0/684493894 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a74075480 con 0x7f3a741f4bf0 2023-11-24T06:08:13.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.485+0000 7f3a717fa700 1 -- 172.21.15.91:0/684493894 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3a68014440 con 0x7f3a741f4bf0 2023-11-24T06:08:13.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.485+0000 7f3a717fa700 1 --2- 172.21.15.91:0/684493894 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3a5806aed0 0x7f3a5806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.485+0000 7f3a73fff700 1 --2- 172.21.15.91:0/684493894 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3a5806aed0 0x7f3a5806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.485+0000 7f3a717fa700 1 -- 172.21.15.91:0/684493894 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a68089d40 con 0x7f3a741f4bf0 2023-11-24T06:08:13.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.485+0000 7f3a73fff700 1 --2- 172.21.15.91:0/684493894 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3a5806aed0 0x7f3a5806d390 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3a740b7b40 tx=0x7f3a60008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:13.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.489+0000 7f3a717fa700 1 -- 172.21.15.91:0/684493894 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a68054f60 con 0x7f3a741f4bf0 2023-11-24T06:08:13.683 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.679+0000 7f3a7b115700 1 -- 172.21.15.91:0/684493894 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3a740b25a0 con 0x7f3a741f4bf0 2023-11-24T06:08:13.684 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.680+0000 7f3a717fa700 1 -- 172.21.15.91:0/684493894 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3a68022020 con 0x7f3a741f4bf0 2023-11-24T06:08:13.684 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:08:13.686 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.682+0000 7f3a7b115700 1 -- 172.21.15.91:0/684493894 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3a5806aed0 msgr2=0x7f3a5806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.686 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.682+0000 7f3a7b115700 1 --2- 172.21.15.91:0/684493894 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3a5806aed0 0x7f3a5806d390 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3a740b7b40 tx=0x7f3a60008040 comp rx=0 tx=0).stop 2023-11-24T06:08:13.686 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.682+0000 7f3a7b115700 1 -- 172.21.15.91:0/684493894 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3a741f4bf0 msgr2=0x7f3a7415e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.686 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.682+0000 7f3a7b115700 1 --2- 172.21.15.91:0/684493894 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3a741f4bf0 0x7f3a7415e470 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3a68011aa0 tx=0x7f3a68011e60 comp rx=0 tx=0).stop 2023-11-24T06:08:13.686 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.682+0000 7f3a7b115700 1 -- 172.21.15.91:0/684493894 shutdown_connections 2023-11-24T06:08:13.687 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.682+0000 7f3a7b115700 1 --2- 172.21.15.91:0/684493894 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3a5806aed0 0x7f3a5806d390 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.687 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.682+0000 7f3a7b115700 1 --2- 172.21.15.91:0/684493894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3a7415e9b0 0x7f3a740c91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.687 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.682+0000 7f3a7b115700 1 --2- 172.21.15.91:0/684493894 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3a741f4bf0 0x7f3a7415e470 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.687 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.682+0000 7f3a7b115700 1 --2- 172.21.15.91:0/684493894 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3a7414c2e0 0x7f3a740c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.687 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.682+0000 7f3a7b115700 1 -- 172.21.15.91:0/684493894 >> 172.21.15.91:0/684493894 conn(0x7f3a740b3720 msgr2=0x7f3a740cf810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:13.687 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.682+0000 7f3a7b115700 1 -- 172.21.15.91:0/684493894 shutdown_connections 2023-11-24T06:08:13.687 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.682+0000 7f3a7b115700 1 -- 172.21.15.91:0/684493894 wait complete. 2023-11-24T06:08:13.698 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:08:13.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.817+0000 7f2758bcb700 1 Processor -- start 2023-11-24T06:08:13.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.817+0000 7f2758bcb700 1 -- start start 2023-11-24T06:08:13.832 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.818+0000 7f2758bcb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2754072a00 0x7f275406db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.832 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.818+0000 7f2758bcb700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27540d0b80 con 0x7f275406e0a0 2023-11-24T06:08:13.832 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.818+0000 7f2758bcb700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f275415ed60 con 0x7f27540b4090 2023-11-24T06:08:13.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.818+0000 7f2758bcb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f275415f040 con 0x7f2754072a00 2023-11-24T06:08:13.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.818+0000 7f2752d9d700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f275406e0a0 0x7f27540d1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53286/0 (socket says 172.21.15.91:53286) 2023-11-24T06:08:13.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.818+0000 7f2752d9d700 1 -- 172.21.15.91:0/3017876742 learned_addr learned my addr 172.21.15.91:0/3017876742 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:13.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.818+0000 7f2751d9b700 1 --2- 172.21.15.91:0/3017876742 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2754072a00 0x7f275406db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.818+0000 7f275159a700 1 -- 172.21.15.91:0/3017876742 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247599779 0 0) 0x7f27540d0b80 con 0x7f275406e0a0 2023-11-24T06:08:13.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.818+0000 7f275159a700 1 -- 172.21.15.91:0/3017876742 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f273c0035b0 con 0x7f275406e0a0 2023-11-24T06:08:13.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.818+0000 7f275159a700 1 -- 172.21.15.91:0/3017876742 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3943349568 0 0) 0x7f273c0035b0 con 0x7f275406e0a0 2023-11-24T06:08:13.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.818+0000 7f275159a700 1 -- 172.21.15.91:0/3017876742 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2754072a00 msgr2=0x7f275406db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:13.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.818+0000 7f275159a700 1 --2- 172.21.15.91:0/3017876742 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2754072a00 0x7f275406db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.818+0000 7f275159a700 1 -- 172.21.15.91:0/3017876742 >> v1:172.21.15.94:6789/0 conn(0x7f27540b4090 legacy=0x7f275415e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.818+0000 7f275159a700 1 -- 172.21.15.91:0/3017876742 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275415f320 con 0x7f275406e0a0 2023-11-24T06:08:13.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.818+0000 7f275159a700 1 -- 172.21.15.91:0/3017876742 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f274c0019f0 con 0x7f275406e0a0 2023-11-24T06:08:13.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f275159a700 1 -- 172.21.15.91:0/3017876742 >> v1:172.21.15.91:6789/0 conn(0x7f275406e0a0 legacy=0x7f27540d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.835 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f275159a700 1 --2- 172.21.15.91:0/3017876742 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2754072a00 0x7f273c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.835 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f275159a700 1 --2- 172.21.15.91:0/3017876742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f273c003fb0 0x7f273c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.835 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f275159a700 1 --2- 172.21.15.91:0/3017876742 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f273c004d90 0x7f273c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.835 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f275159a700 1 -- 172.21.15.91:0/3017876742 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f275415f040 con 0x7f273c004d90 2023-11-24T06:08:13.835 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f275159a700 1 -- 172.21.15.91:0/3017876742 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273c00bba0 con 0x7f273c003fb0 2023-11-24T06:08:13.835 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f275159a700 1 -- 172.21.15.91:0/3017876742 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f273c00bd60 con 0x7f2754072a00 2023-11-24T06:08:13.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f2752d9d700 1 --2- 172.21.15.91:0/3017876742 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f273c004d90 0x7f273c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f2751d9b700 1 --2- 172.21.15.91:0/3017876742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f273c003fb0 0x7f273c004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f275259c700 1 --2- 172.21.15.91:0/3017876742 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2754072a00 0x7f273c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f2752d9d700 1 -- 172.21.15.91:0/3017876742 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2754072a00 msgr2=0x7f273c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:13.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f2752d9d700 1 --2- 172.21.15.91:0/3017876742 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2754072a00 0x7f273c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f2752d9d700 1 -- 172.21.15.91:0/3017876742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f273c003fb0 msgr2=0x7f273c004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:13.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f2752d9d700 1 --2- 172.21.15.91:0/3017876742 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f273c003fb0 0x7f273c004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f2752d9d700 1 -- 172.21.15.91:0/3017876742 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273c0035b0 con 0x7f273c004d90 2023-11-24T06:08:13.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f2752d9d700 1 --2- 172.21.15.91:0/3017876742 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f273c004d90 0x7f273c005210 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f274c00a830 tx=0x7f273c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b6046f92e3b959 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:13.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f275159a700 1 -- 172.21.15.91:0/3017876742 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c00a2f0 con 0x7f273c004d90 2023-11-24T06:08:13.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f275159a700 1 -- 172.21.15.91:0/3017876742 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274c012600 con 0x7f273c004d90 2023-11-24T06:08:13.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.819+0000 7f275159a700 1 -- 172.21.15.91:0/3017876742 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274c0129b0 con 0x7f273c004d90 2023-11-24T06:08:13.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.820+0000 7f2758bcb700 1 -- 172.21.15.91:0/3017876742 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f273c004d90 msgr2=0x7f273c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.820+0000 7f2758bcb700 1 --2- 172.21.15.91:0/3017876742 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f273c004d90 0x7f273c005210 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f274c00a830 tx=0x7f273c005880 comp rx=0 tx=0).stop 2023-11-24T06:08:13.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.820+0000 7f2752d9d700 1 -- 172.21.15.91:0/3017876742 reap_dead start 2023-11-24T06:08:13.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.820+0000 7f2758bcb700 1 -- 172.21.15.91:0/3017876742 shutdown_connections 2023-11-24T06:08:13.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.820+0000 7f2758bcb700 1 -- 172.21.15.91:0/3017876742 >> 172.21.15.91:0/3017876742 conn(0x7f27540b2740 msgr2=0x7f27540b2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:13.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.820+0000 7f2758bcb700 1 -- 172.21.15.91:0/3017876742 shutdown_connections 2023-11-24T06:08:13.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.820+0000 7f2758bcb700 1 -- 172.21.15.91:0/3017876742 wait complete. 2023-11-24T06:08:13.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.821+0000 7f2758bcb700 1 Processor -- start 2023-11-24T06:08:13.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.821+0000 7f2758bcb700 1 -- start start 2023-11-24T06:08:13.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f2758bcb700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f273c004d90 0x7f27540c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f2758bcb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2754147580 0x7f2754148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f2758bcb700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2754146c50 0x7f27540b1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f2758bcb700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2754147a00 con 0x7f273c004d90 2023-11-24T06:08:13.840 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f2758bcb700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2754147cd0 con 0x7f2754146c50 2023-11-24T06:08:13.840 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f2758bcb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2754147fb0 con 0x7f2754147580 2023-11-24T06:08:13.840 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f275259c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f273c004d90 0x7f27540c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.840 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f275259c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f273c004d90 0x7f27540c63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58360/0 (socket says 172.21.15.91:58360) 2023-11-24T06:08:13.840 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f275259c700 1 -- 172.21.15.91:0/3687210530 learned_addr learned my addr 172.21.15.91:0/3687210530 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:13.840 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f2752d9d700 1 --2- 172.21.15.91:0/3687210530 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2754146c50 0x7f27540b1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.840 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f2751d9b700 1 --2- 172.21.15.91:0/3687210530 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2754147580 0x7f2754148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.841 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f275259c700 1 -- 172.21.15.91:0/3687210530 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2754147580 msgr2=0x7f2754148d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.841 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f275259c700 1 --2- 172.21.15.91:0/3687210530 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2754147580 0x7f2754148d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.841 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f275259c700 1 -- 172.21.15.91:0/3687210530 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2754146c50 msgr2=0x7f27540b1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:13.841 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f275259c700 1 --2- 172.21.15.91:0/3687210530 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2754146c50 0x7f27540b1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:13.841 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f275259c700 1 -- 172.21.15.91:0/3687210530 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2754148290 con 0x7f273c004d90 2023-11-24T06:08:13.841 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f2751d9b700 1 --2- 172.21.15.91:0/3687210530 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2754147580 0x7f2754148d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:13.842 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f275259c700 1 --2- 172.21.15.91:0/3687210530 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f273c004d90 0x7f27540c63c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f274400f820 tx=0x7f274400fb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:13.842 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.822+0000 7f2742ffd700 1 -- 172.21.15.91:0/3687210530 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27440052e0 con 0x7f273c004d90 2023-11-24T06:08:13.842 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.823+0000 7f2742ffd700 1 -- 172.21.15.91:0/3687210530 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2744005440 con 0x7f273c004d90 2023-11-24T06:08:13.842 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.823+0000 7f2758bcb700 1 -- 172.21.15.91:0/3687210530 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2754148570 con 0x7f273c004d90 2023-11-24T06:08:13.842 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.823+0000 7f2742ffd700 1 -- 172.21.15.91:0/3687210530 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27440038c0 con 0x7f273c004d90 2023-11-24T06:08:13.842 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.823+0000 7f2758bcb700 1 -- 172.21.15.91:0/3687210530 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27541492b0 con 0x7f273c004d90 2023-11-24T06:08:13.842 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.823+0000 7f2758bcb700 1 -- 172.21.15.91:0/3687210530 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2754075480 con 0x7f273c004d90 2023-11-24T06:08:13.842 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.824+0000 7f2742ffd700 1 -- 172.21.15.91:0/3687210530 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2744003a20 con 0x7f273c004d90 2023-11-24T06:08:13.842 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.824+0000 7f2742ffd700 1 --2- 172.21.15.91:0/3687210530 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f273406aed0 0x7f273406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:13.842 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.824+0000 7f2751d9b700 1 --2- 172.21.15.91:0/3687210530 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f273406aed0 0x7f273406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:13.843 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.824+0000 7f2742ffd700 1 -- 172.21.15.91:0/3687210530 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2744089390 con 0x7f273c004d90 2023-11-24T06:08:13.843 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.825+0000 7f2751d9b700 1 --2- 172.21.15.91:0/3687210530 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f273406aed0 0x7f273406d390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f27540b2550 tx=0x7f2748008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:13.843 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:13.829+0000 7f2742ffd700 1 -- 172.21.15.91:0/3687210530 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2744054440 con 0x7f273c004d90 2023-11-24T06:08:14.026 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.022+0000 7f2758bcb700 1 -- 172.21.15.91:0/3687210530 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2754072650 con 0x7f273c004d90 2023-11-24T06:08:14.026 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.023+0000 7f2742ffd700 1 -- 172.21.15.91:0/3687210530 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2744057a60 con 0x7f273c004d90 2023-11-24T06:08:14.027 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 32 2023-11-24T06:08:14.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.025+0000 7f2758bcb700 1 -- 172.21.15.91:0/3687210530 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f273406aed0 msgr2=0x7f273406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.025+0000 7f2758bcb700 1 --2- 172.21.15.91:0/3687210530 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f273406aed0 0x7f273406d390 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f27540b2550 tx=0x7f2748008040 comp rx=0 tx=0).stop 2023-11-24T06:08:14.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.025+0000 7f2758bcb700 1 -- 172.21.15.91:0/3687210530 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f273c004d90 msgr2=0x7f27540c63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.025+0000 7f2758bcb700 1 --2- 172.21.15.91:0/3687210530 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f273c004d90 0x7f27540c63c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f274400f820 tx=0x7f274400fb30 comp rx=0 tx=0).stop 2023-11-24T06:08:14.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.025+0000 7f2758bcb700 1 -- 172.21.15.91:0/3687210530 shutdown_connections 2023-11-24T06:08:14.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.025+0000 7f2758bcb700 1 --2- 172.21.15.91:0/3687210530 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f273406aed0 0x7f273406d390 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.025+0000 7f2758bcb700 1 --2- 172.21.15.91:0/3687210530 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2754146c50 0x7f27540b1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.025+0000 7f2758bcb700 1 --2- 172.21.15.91:0/3687210530 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f273c004d90 0x7f27540c63c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.025+0000 7f2758bcb700 1 --2- 172.21.15.91:0/3687210530 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2754147580 0x7f2754148d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.025+0000 7f2758bcb700 1 -- 172.21.15.91:0/3687210530 >> 172.21.15.91:0/3687210530 conn(0x7f27540b2740 msgr2=0x7f275406afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:14.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.025+0000 7f2758bcb700 1 -- 172.21.15.91:0/3687210530 shutdown_connections 2023-11-24T06:08:14.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.025+0000 7f2758bcb700 1 -- 172.21.15.91:0/3687210530 wait complete. 2023-11-24T06:08:14.041 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:08:14.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.160+0000 7f7844385700 1 Processor -- start 2023-11-24T06:08:14.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.160+0000 7f7844385700 1 -- start start 2023-11-24T06:08:14.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.161+0000 7f7844385700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f783c1564e0 0x7f783c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.161+0000 7f7844385700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f783c06e770 con 0x7f783c15cca0 2023-11-24T06:08:14.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.161+0000 7f7844385700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f783c06ea50 con 0x7f783c06afd0 2023-11-24T06:08:14.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.161+0000 7f7844385700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f783c06ed30 con 0x7f783c1564e0 2023-11-24T06:08:14.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.161+0000 7f7841920700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f783c1564e0 0x7f783c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.161+0000 7f7842922700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f783c15cca0 0x7f783c0cf2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53306/0 (socket says 172.21.15.91:53306) 2023-11-24T06:08:14.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.161+0000 7f7842922700 1 -- 172.21.15.91:0/3076568706 learned_addr learned my addr 172.21.15.91:0/3076568706 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:14.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.161+0000 7f7841920700 1 -- 172.21.15.91:0/3076568706 >> v1:172.21.15.94:6789/0 conn(0x7f783c06afd0 legacy=0x7f783c06b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:14.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.161+0000 7f7841920700 1 -- 172.21.15.91:0/3076568706 >> v1:172.21.15.91:6789/0 conn(0x7f783c15cca0 legacy=0x7f783c0cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.161+0000 7f7841920700 1 -- 172.21.15.91:0/3076568706 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783c06f010 con 0x7f783c1564e0 2023-11-24T06:08:14.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.161+0000 7f784111f700 1 -- 172.21.15.91:0/3076568706 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114000663 0 0) 0x7f783c06e770 con 0x7f783c15cca0 2023-11-24T06:08:14.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.161+0000 7f7841920700 1 --2- 172.21.15.91:0/3076568706 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f783c1564e0 0x7f783c06e0e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f782c009840 tx=0x7f782c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a99638bde4785c11 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:14.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.161+0000 7f784111f700 1 -- 172.21.15.91:0/3076568706 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782c004230 con 0x7f783c1564e0 2023-11-24T06:08:14.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.161+0000 7f784111f700 1 -- 172.21.15.91:0/3076568706 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f782c004390 con 0x7f783c1564e0 2023-11-24T06:08:14.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.161+0000 7f784111f700 1 -- 172.21.15.91:0/3076568706 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f782c0036a0 con 0x7f783c1564e0 2023-11-24T06:08:14.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.162+0000 7f7844385700 1 -- 172.21.15.91:0/3076568706 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f783c1564e0 msgr2=0x7f783c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.162+0000 7f7844385700 1 --2- 172.21.15.91:0/3076568706 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f783c1564e0 0x7f783c06e0e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f782c009840 tx=0x7f782c009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:14.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.162+0000 7f7844385700 1 -- 172.21.15.91:0/3076568706 shutdown_connections 2023-11-24T06:08:14.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.162+0000 7f7844385700 1 --2- 172.21.15.91:0/3076568706 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f783c1564e0 0x7f783c06e0e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.162+0000 7f7844385700 1 -- 172.21.15.91:0/3076568706 >> 172.21.15.91:0/3076568706 conn(0x7f783c0b27b0 msgr2=0x7f783c0b2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:14.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.162+0000 7f7844385700 1 -- 172.21.15.91:0/3076568706 shutdown_connections 2023-11-24T06:08:14.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.162+0000 7f7844385700 1 -- 172.21.15.91:0/3076568706 wait complete. 2023-11-24T06:08:14.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.163+0000 7f7844385700 1 Processor -- start 2023-11-24T06:08:14.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.163+0000 7f7844385700 1 -- start start 2023-11-24T06:08:14.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7844385700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f783c06afd0 0x7f783c14df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7844385700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f783c1564e0 0x7f783c0d0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7844385700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f783c15cca0 0x7f783c0d1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7844385700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783c148e30 con 0x7f783c15cca0 2023-11-24T06:08:14.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7844385700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783c14bd20 con 0x7f783c06afd0 2023-11-24T06:08:14.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7844385700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f783c14b110 con 0x7f783c1564e0 2023-11-24T06:08:14.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7842922700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f783c15cca0 0x7f783c0d1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7842922700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f783c15cca0 0x7f783c0d1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58378/0 (socket says 172.21.15.91:58378) 2023-11-24T06:08:14.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7842922700 1 -- 172.21.15.91:0/563932000 learned_addr learned my addr 172.21.15.91:0/563932000 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:14.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7842121700 1 --2- 172.21.15.91:0/563932000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f783c06afd0 0x7f783c14df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7841920700 1 --2- 172.21.15.91:0/563932000 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f783c1564e0 0x7f783c0d0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7842922700 1 -- 172.21.15.91:0/563932000 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f783c1564e0 msgr2=0x7f783c0d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7842922700 1 --2- 172.21.15.91:0/563932000 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f783c1564e0 0x7f783c0d0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7842922700 1 -- 172.21.15.91:0/563932000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f783c06afd0 msgr2=0x7f783c14df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7842922700 1 --2- 172.21.15.91:0/563932000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f783c06afd0 0x7f783c14df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7842922700 1 -- 172.21.15.91:0/563932000 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c0094a0 con 0x7f783c15cca0 2023-11-24T06:08:14.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7841920700 1 --2- 172.21.15.91:0/563932000 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f783c1564e0 0x7f783c0d0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:14.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.164+0000 7f7842922700 1 --2- 172.21.15.91:0/563932000 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f783c15cca0 0x7f783c0d1000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f783400cb60 tx=0x7f783400ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:14.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.165+0000 7f7832ffd700 1 -- 172.21.15.91:0/563932000 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783400aaa0 con 0x7f783c15cca0 2023-11-24T06:08:14.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.165+0000 7f7832ffd700 1 -- 172.21.15.91:0/563932000 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783400ac00 con 0x7f783c15cca0 2023-11-24T06:08:14.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.165+0000 7f7844385700 1 -- 172.21.15.91:0/563932000 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f783c14b3f0 con 0x7f783c15cca0 2023-11-24T06:08:14.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.165+0000 7f7832ffd700 1 -- 172.21.15.91:0/563932000 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78340070d0 con 0x7f783c15cca0 2023-11-24T06:08:14.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.165+0000 7f7844385700 1 -- 172.21.15.91:0/563932000 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f783c14b9b0 con 0x7f783c15cca0 2023-11-24T06:08:14.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.166+0000 7f7844385700 1 -- 172.21.15.91:0/563932000 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f783c075480 con 0x7f783c15cca0 2023-11-24T06:08:14.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.166+0000 7f7832ffd700 1 -- 172.21.15.91:0/563932000 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7834007230 con 0x7f783c15cca0 2023-11-24T06:08:14.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.166+0000 7f7832ffd700 1 --2- 172.21.15.91:0/563932000 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f782006aed0 0x7f782006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.166+0000 7f7832ffd700 1 -- 172.21.15.91:0/563932000 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7834089360 con 0x7f783c15cca0 2023-11-24T06:08:14.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.166+0000 7f7842121700 1 --2- 172.21.15.91:0/563932000 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f782006aed0 0x7f782006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.167+0000 7f7842121700 1 --2- 172.21.15.91:0/563932000 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f782006aed0 0x7f782006d390 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f783c14fa10 tx=0x7f7838007ec0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:14.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.171+0000 7f7832ffd700 1 -- 172.21.15.91:0/563932000 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7834054410 con 0x7f783c15cca0 2023-11-24T06:08:14.365 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.361+0000 7f7844385700 1 -- 172.21.15.91:0/563932000 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f783c14be60 con 0x7f783c15cca0 2023-11-24T06:08:14.365 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.361+0000 7f7832ffd700 1 -- 172.21.15.91:0/563932000 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7834057a30 con 0x7f783c15cca0 2023-11-24T06:08:14.365 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 64 2023-11-24T06:08:14.367 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.363+0000 7f7844385700 1 -- 172.21.15.91:0/563932000 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f782006aed0 msgr2=0x7f782006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.367 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.363+0000 7f7844385700 1 --2- 172.21.15.91:0/563932000 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f782006aed0 0x7f782006d390 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f783c14fa10 tx=0x7f7838007ec0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.367 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.363+0000 7f7844385700 1 -- 172.21.15.91:0/563932000 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f783c15cca0 msgr2=0x7f783c0d1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.363+0000 7f7844385700 1 --2- 172.21.15.91:0/563932000 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f783c15cca0 0x7f783c0d1000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f783400cb60 tx=0x7f783400ce70 comp rx=0 tx=0).stop 2023-11-24T06:08:14.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.363+0000 7f7844385700 1 -- 172.21.15.91:0/563932000 shutdown_connections 2023-11-24T06:08:14.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.363+0000 7f7844385700 1 --2- 172.21.15.91:0/563932000 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f782006aed0 0x7f782006d390 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.363+0000 7f7844385700 1 --2- 172.21.15.91:0/563932000 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f783c06afd0 0x7f783c14df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.363+0000 7f7844385700 1 --2- 172.21.15.91:0/563932000 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f783c15cca0 0x7f783c0d1000 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.363+0000 7f7844385700 1 --2- 172.21.15.91:0/563932000 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f783c1564e0 0x7f783c0d0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.363+0000 7f7844385700 1 -- 172.21.15.91:0/563932000 >> 172.21.15.91:0/563932000 conn(0x7f783c0b27b0 msgr2=0x7f783c15c070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:14.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.363+0000 7f7844385700 1 -- 172.21.15.91:0/563932000 shutdown_connections 2023-11-24T06:08:14.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.363+0000 7f7844385700 1 -- 172.21.15.91:0/563932000 wait complete. 2023-11-24T06:08:14.379 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:08:14.499 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.495+0000 7fcee63d3700 1 Processor -- start 2023-11-24T06:08:14.499 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.495+0000 7fcee63d3700 1 -- start start 2023-11-24T06:08:14.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.496+0000 7fcee63d3700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fcee006b620 0x7fcee00b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.496+0000 7fcee63d3700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcee00b5ab0 0x7fcee00b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.496+0000 7fcee63d3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee01478c0 0x7fcee0161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.496+0000 7fcee63d3700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee0071f50 con 0x7fcee006b620 2023-11-24T06:08:14.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.496+0000 7fcee63d3700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee0072230 con 0x7fcee00b5ab0 2023-11-24T06:08:14.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.496+0000 7fcee63d3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee0072510 con 0x7fcee01478c0 2023-11-24T06:08:14.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.496+0000 7fcedffff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fcee006b620 0x7fcee00b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.496+0000 7fcedffff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fcee006b620 0x7fcee00b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58382/0 (socket says 172.21.15.91:58382) 2023-11-24T06:08:14.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.496+0000 7fcedffff700 1 -- 172.21.15.91:0/2878278764 learned_addr learned my addr 172.21.15.91:0/2878278764 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:14.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.496+0000 7fcedf7fe700 1 --2- 172.21.15.91:0/2878278764 >> v2:172.21.15.94:3300/0 conn(0x7fcee00b5ab0 0x7fcee00b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.496+0000 7fcee4970700 1 --2- 172.21.15.91:0/2878278764 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee01478c0 0x7fcee0161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcedffff700 1 -- 172.21.15.91:0/2878278764 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee01478c0 msgr2=0x7fcee0161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:14.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcedffff700 1 --2- 172.21.15.91:0/2878278764 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee01478c0 0x7fcee0161ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcedffff700 1 -- 172.21.15.91:0/2878278764 >> v2:172.21.15.94:3300/0 conn(0x7fcee00b5ab0 msgr2=0x7fcee00b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:14.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcedffff700 1 --2- 172.21.15.91:0/2878278764 >> v2:172.21.15.94:3300/0 conn(0x7fcee00b5ab0 0x7fcee00b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcedffff700 1 -- 172.21.15.91:0/2878278764 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee00727f0 con 0x7fcee006b620 2023-11-24T06:08:14.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcedffff700 1 --2- 172.21.15.91:0/2878278764 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcee006b620 0x7fcee00b3a40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcee006d6f0 tx=0x7fced4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2899c6506d1a885 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:14.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcedeffd700 1 -- 172.21.15.91:0/2878278764 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced400b870 con 0x7fcee006b620 2023-11-24T06:08:14.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcedeffd700 1 -- 172.21.15.91:0/2878278764 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced400b9d0 con 0x7fcee006b620 2023-11-24T06:08:14.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcedeffd700 1 -- 172.21.15.91:0/2878278764 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced400bd60 con 0x7fcee006b620 2023-11-24T06:08:14.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcee63d3700 1 -- 172.21.15.91:0/2878278764 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcee006b620 msgr2=0x7fcee00b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcee63d3700 1 --2- 172.21.15.91:0/2878278764 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcee006b620 0x7fcee00b3a40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcee006d6f0 tx=0x7fced4009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcee63d3700 1 -- 172.21.15.91:0/2878278764 shutdown_connections 2023-11-24T06:08:14.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcee63d3700 1 --2- 172.21.15.91:0/2878278764 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee01478c0 0x7fcee0161ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcee63d3700 1 --2- 172.21.15.91:0/2878278764 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcee006b620 0x7fcee00b3a40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcee63d3700 1 --2- 172.21.15.91:0/2878278764 >> v2:172.21.15.94:3300/0 conn(0x7fcee00b5ab0 0x7fcee00b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcee63d3700 1 -- 172.21.15.91:0/2878278764 >> 172.21.15.91:0/2878278764 conn(0x7fcee00b2740 msgr2=0x7fcee00b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:14.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.497+0000 7fcee63d3700 1 -- 172.21.15.91:0/2878278764 shutdown_connections 2023-11-24T06:08:14.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.498+0000 7fcee63d3700 1 -- 172.21.15.91:0/2878278764 wait complete. 2023-11-24T06:08:14.506 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.499+0000 7fcee63d3700 1 Processor -- start 2023-11-24T06:08:14.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.499+0000 7fcee63d3700 1 -- start start 2023-11-24T06:08:14.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.499+0000 7fcee63d3700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcee006b620 0x7fcee0146550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcee63d3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee00b5ab0 0x7fcee01dca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcee63d3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee01478c0 0x7fcee01dcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcee63d3700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee01e4120 con 0x7fcee006b620 2023-11-24T06:08:14.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcee63d3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee01e43c0 con 0x7fcee01478c0 2023-11-24T06:08:14.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcee63d3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee01e46a0 con 0x7fcee00b5ab0 2023-11-24T06:08:14.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcedffff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcee006b620 0x7fcee0146550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcedffff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcee006b620 0x7fcee0146550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58388/0 (socket says 172.21.15.91:58388) 2023-11-24T06:08:14.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcedffff700 1 -- 172.21.15.91:0/3901807539 learned_addr learned my addr 172.21.15.91:0/3901807539 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:14.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcedf7fe700 1 --2- 172.21.15.91:0/3901807539 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee00b5ab0 0x7fcee01dca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcee4970700 1 --2- 172.21.15.91:0/3901807539 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee01478c0 0x7fcee01dcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcedffff700 1 -- 172.21.15.91:0/3901807539 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee00b5ab0 msgr2=0x7fcee01dca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcedffff700 1 --2- 172.21.15.91:0/3901807539 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee00b5ab0 0x7fcee01dca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcedffff700 1 -- 172.21.15.91:0/3901807539 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee01478c0 msgr2=0x7fcee01dcfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcedffff700 1 --2- 172.21.15.91:0/3901807539 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee01478c0 0x7fcee01dcfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcedffff700 1 -- 172.21.15.91:0/3901807539 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced4009550 con 0x7fcee006b620 2023-11-24T06:08:14.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcee4970700 1 --2- 172.21.15.91:0/3901807539 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee01478c0 0x7fcee01dcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:14.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcedffff700 1 --2- 172.21.15.91:0/3901807539 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcee006b620 0x7fcee0146550 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fced4000c00 tx=0x7fced4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:14.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcedcff9700 1 -- 172.21.15.91:0/3901807539 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4004540 con 0x7fcee006b620 2023-11-24T06:08:14.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcedcff9700 1 -- 172.21.15.91:0/3901807539 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced40036a0 con 0x7fcee006b620 2023-11-24T06:08:14.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcee63d3700 1 -- 172.21.15.91:0/3901807539 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee01e4980 con 0x7fcee006b620 2023-11-24T06:08:14.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcedcff9700 1 -- 172.21.15.91:0/3901807539 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4003a50 con 0x7fcee006b620 2023-11-24T06:08:14.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.500+0000 7fcee63d3700 1 -- 172.21.15.91:0/3901807539 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee01ddbd0 con 0x7fcee006b620 2023-11-24T06:08:14.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.501+0000 7fcee63d3700 1 -- 172.21.15.91:0/3901807539 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee01e13f0 con 0x7fcee006b620 2023-11-24T06:08:14.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.502+0000 7fcedcff9700 1 -- 172.21.15.91:0/3901807539 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fced4021070 con 0x7fcee006b620 2023-11-24T06:08:14.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.502+0000 7fcedcff9700 1 --2- 172.21.15.91:0/3901807539 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcecc06aed0 0x7fcecc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.502+0000 7fcedcff9700 1 -- 172.21.15.91:0/3901807539 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fced408ac70 con 0x7fcee006b620 2023-11-24T06:08:14.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.502+0000 7fcedf7fe700 1 --2- 172.21.15.91:0/3901807539 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcecc06aed0 0x7fcecc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.502+0000 7fcedf7fe700 1 --2- 172.21.15.91:0/3901807539 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcecc06aed0 0x7fcecc06d390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcee01e3230 tx=0x7fced0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:14.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.506+0000 7fcedcff9700 1 -- 172.21.15.91:0/3901807539 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fced4055db0 con 0x7fcee006b620 2023-11-24T06:08:14.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.741+0000 7fcee63d3700 1 -- 172.21.15.91:0/3901807539 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcee0068670 con 0x7fcee006b620 2023-11-24T06:08:14.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.742+0000 7fcedcff9700 1 -- 172.21.15.91:0/3901807539 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4907 (secure 0 0 0) 0x7fced4015020 con 0x7fcee006b620 2023-11-24T06:08:14.746 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:14.746 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1329219967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1329219967},{"type":"v1","addr":"172.21.15.94:6834","nonce":1329219967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3439524864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3439524864},{"type":"v1","addr":"172.21.15.94:6835","nonce":3439524864}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6836/3390368219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6834","nonce":3390368219},{"type":"v1","addr":"172.21.15.91:6836","nonce":3390368219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:08:00.798681+0000","modified":"2023-11-24T06:08:04.132764+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.91:6837/1644079645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":1644079645},{"type":"v1","addr":"172.21.15.91:6837","nonce":1644079645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T06:08:14.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.744+0000 7fcee63d3700 1 -- 172.21.15.91:0/3901807539 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcecc06aed0 msgr2=0x7fcecc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.744+0000 7fcee63d3700 1 --2- 172.21.15.91:0/3901807539 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcecc06aed0 0x7fcecc06d390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcee01e3230 tx=0x7fced0008040 comp rx=0 tx=0).stop 2023-11-24T06:08:14.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.744+0000 7fcee63d3700 1 -- 172.21.15.91:0/3901807539 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcee006b620 msgr2=0x7fcee0146550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.744+0000 7fcee63d3700 1 --2- 172.21.15.91:0/3901807539 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcee006b620 0x7fcee0146550 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fced4000c00 tx=0x7fced4004750 comp rx=0 tx=0).stop 2023-11-24T06:08:14.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.744+0000 7fcee63d3700 1 -- 172.21.15.91:0/3901807539 shutdown_connections 2023-11-24T06:08:14.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.744+0000 7fcee63d3700 1 --2- 172.21.15.91:0/3901807539 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcecc06aed0 0x7fcecc06d390 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.744+0000 7fcee63d3700 1 --2- 172.21.15.91:0/3901807539 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcee01478c0 0x7fcee01dcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.744+0000 7fcee63d3700 1 --2- 172.21.15.91:0/3901807539 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcee006b620 0x7fcee0146550 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.744+0000 7fcee63d3700 1 --2- 172.21.15.91:0/3901807539 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcee00b5ab0 0x7fcee01dca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.744+0000 7fcee63d3700 1 -- 172.21.15.91:0/3901807539 >> 172.21.15.91:0/3901807539 conn(0x7fcee00b2740 msgr2=0x7fcee01dc0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:14.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.744+0000 7fcee63d3700 1 -- 172.21.15.91:0/3901807539 shutdown_connections 2023-11-24T06:08:14.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.744+0000 7fcee63d3700 1 -- 172.21.15.91:0/3901807539 wait complete. 2023-11-24T06:08:14.749 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 8 2023-11-24T06:08:14.760 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:14.880 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.876+0000 7f52aaa62700 1 Processor -- start 2023-11-24T06:08:14.880 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.876+0000 7f52aaa62700 1 -- start start 2023-11-24T06:08:14.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.877+0000 7f52aaa62700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f52a415e230 0x7f52a40cf060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.877+0000 7f52aaa62700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a4162130 con 0x7f52a4161bc0 2023-11-24T06:08:14.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.877+0000 7f52aaa62700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a4162410 con 0x7f52a406b6e0 2023-11-24T06:08:14.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.877+0000 7f52aaa62700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a41626f0 con 0x7f52a415e230 2023-11-24T06:08:14.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.877+0000 7f52a37fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f52a4161bc0 0x7f52a406afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53330/0 (socket says 172.21.15.91:53330) 2023-11-24T06:08:14.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.877+0000 7f52a37fe700 1 -- 172.21.15.91:0/130297671 learned_addr learned my addr 172.21.15.91:0/130297671 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:14.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.877+0000 7f52a3fff700 1 --2- 172.21.15.91:0/130297671 >> v2:172.21.15.94:3300/0 conn(0x7f52a415e230 0x7f52a40cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.877+0000 7f52a2ffd700 1 -- 172.21.15.91:0/130297671 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2390916101 0 0) 0x7f52a4162130 con 0x7f52a4161bc0 2023-11-24T06:08:14.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.877+0000 7f52a2ffd700 1 -- 172.21.15.91:0/130297671 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f528c0035b0 con 0x7f52a4161bc0 2023-11-24T06:08:14.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.877+0000 7f52a3fff700 1 -- 172.21.15.91:0/130297671 >> v1:172.21.15.94:6789/0 conn(0x7f52a406b6e0 legacy=0x7f52a40cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.877+0000 7f52a3fff700 1 -- 172.21.15.91:0/130297671 >> v1:172.21.15.91:6789/0 conn(0x7f52a4161bc0 legacy=0x7f52a406afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.877+0000 7f52a3fff700 1 -- 172.21.15.91:0/130297671 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a41629d0 con 0x7f52a415e230 2023-11-24T06:08:14.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.878+0000 7f52a3fff700 1 --2- 172.21.15.91:0/130297671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52a415e230 0x7f52a40cf060 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5298009840 tx=0x7f5298009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c0dc8600986605c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:14.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.878+0000 7f52a2ffd700 1 -- 172.21.15.91:0/130297671 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298011830 con 0x7f52a415e230 2023-11-24T06:08:14.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.878+0000 7f52a2ffd700 1 -- 172.21.15.91:0/130297671 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5298011990 con 0x7f52a415e230 2023-11-24T06:08:14.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.878+0000 7f52a2ffd700 1 -- 172.21.15.91:0/130297671 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298011d00 con 0x7f52a415e230 2023-11-24T06:08:14.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.878+0000 7f52aaa62700 1 -- 172.21.15.91:0/130297671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52a415e230 msgr2=0x7f52a40cf060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.878+0000 7f52aaa62700 1 --2- 172.21.15.91:0/130297671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52a415e230 0x7f52a40cf060 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5298009840 tx=0x7f5298009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:14.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.878+0000 7f52aaa62700 1 -- 172.21.15.91:0/130297671 shutdown_connections 2023-11-24T06:08:14.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.878+0000 7f52aaa62700 1 --2- 172.21.15.91:0/130297671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52a415e230 0x7f52a40cf060 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.878+0000 7f52aaa62700 1 -- 172.21.15.91:0/130297671 >> 172.21.15.91:0/130297671 conn(0x7f52a40d0a80 msgr2=0x7f52a40d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:14.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.878+0000 7f52aaa62700 1 -- 172.21.15.91:0/130297671 shutdown_connections 2023-11-24T06:08:14.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.878+0000 7f52aaa62700 1 -- 172.21.15.91:0/130297671 wait complete. 2023-11-24T06:08:14.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.880+0000 7f52aaa62700 1 Processor -- start 2023-11-24T06:08:14.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.880+0000 7f52aaa62700 1 -- start start 2023-11-24T06:08:14.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.880+0000 7f52aaa62700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52a406b6e0 0x7f52a4145e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.880+0000 7f52aaa62700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52a415e230 0x7f52a406e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.880+0000 7f52aaa62700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f52a4161bc0 0x7f52a40b1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.880+0000 7f52aaa62700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a4146640 con 0x7f52a4161bc0 2023-11-24T06:08:14.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.880+0000 7f52aaa62700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a406ee50 con 0x7f52a406b6e0 2023-11-24T06:08:14.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.880+0000 7f52aaa62700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a406f130 con 0x7f52a415e230 2023-11-24T06:08:14.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52a8fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f52a4161bc0 0x7f52a40b1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52a8fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f52a4161bc0 0x7f52a40b1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58404/0 (socket says 172.21.15.91:58404) 2023-11-24T06:08:14.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52a8fff700 1 -- 172.21.15.91:0/260198185 learned_addr learned my addr 172.21.15.91:0/260198185 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:14.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52a3fff700 1 --2- 172.21.15.91:0/260198185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52a406b6e0 0x7f52a4145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52a37fe700 1 --2- 172.21.15.91:0/260198185 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52a415e230 0x7f52a406e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52a8fff700 1 -- 172.21.15.91:0/260198185 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52a415e230 msgr2=0x7f52a406e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52a8fff700 1 --2- 172.21.15.91:0/260198185 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52a415e230 0x7f52a406e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52a8fff700 1 -- 172.21.15.91:0/260198185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52a406b6e0 msgr2=0x7f52a4145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:14.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52a8fff700 1 --2- 172.21.15.91:0/260198185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52a406b6e0 0x7f52a4145e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:14.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52a8fff700 1 -- 172.21.15.91:0/260198185 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52980094a0 con 0x7f52a4161bc0 2023-11-24T06:08:14.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52a37fe700 1 --2- 172.21.15.91:0/260198185 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52a415e230 0x7f52a406e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:14.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52a8fff700 1 --2- 172.21.15.91:0/260198185 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f52a4161bc0 0x7f52a40b1670 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f529c00cbc0 tx=0x7f529c00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:14.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52a0ff9700 1 -- 172.21.15.91:0/260198185 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c003c00 con 0x7f52a4161bc0 2023-11-24T06:08:14.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52a0ff9700 1 -- 172.21.15.91:0/260198185 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529c003d60 con 0x7f52a4161bc0 2023-11-24T06:08:14.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52aaa62700 1 -- 172.21.15.91:0/260198185 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a406f410 con 0x7f52a4161bc0 2023-11-24T06:08:14.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52a0ff9700 1 -- 172.21.15.91:0/260198185 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c00ab40 con 0x7f52a4161bc0 2023-11-24T06:08:14.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.881+0000 7f52aaa62700 1 -- 172.21.15.91:0/260198185 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a406ea90 con 0x7f52a4161bc0 2023-11-24T06:08:14.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.882+0000 7f52aaa62700 1 -- 172.21.15.91:0/260198185 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a4067f60 con 0x7f52a4161bc0 2023-11-24T06:08:14.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.883+0000 7f52a0ff9700 1 -- 172.21.15.91:0/260198185 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f529c00aca0 con 0x7f52a4161bc0 2023-11-24T06:08:14.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.883+0000 7f52a0ff9700 1 --2- 172.21.15.91:0/260198185 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f529006aed0 0x7f529006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:14.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.883+0000 7f52a0ff9700 1 -- 172.21.15.91:0/260198185 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f529c089650 con 0x7f52a4161bc0 2023-11-24T06:08:14.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.883+0000 7f52a3fff700 1 --2- 172.21.15.91:0/260198185 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f529006aed0 0x7f529006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:14.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.883+0000 7f52a3fff700 1 --2- 172.21.15.91:0/260198185 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f529006aed0 0x7f529006d390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5298000c00 tx=0x7f5298008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:14.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:14.887+0000 7f52a0ff9700 1 -- 172.21.15.91:0/260198185 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f529c054700 con 0x7f52a4161bc0 2023-11-24T06:08:15.083 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.079+0000 7f52aaa62700 1 -- 172.21.15.91:0/260198185 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52a4146780 con 0x7f52a4161bc0 2023-11-24T06:08:15.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.080+0000 7f52a0ff9700 1 -- 172.21.15.91:0/260198185 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7f529c057d20 con 0x7f52a4161bc0 2023-11-24T06:08:15.084 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:15.084 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":20,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:02.415612+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:15.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.082+0000 7f52aaa62700 1 -- 172.21.15.91:0/260198185 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f529006aed0 msgr2=0x7f529006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:15.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.082+0000 7f52aaa62700 1 --2- 172.21.15.91:0/260198185 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f529006aed0 0x7f529006d390 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5298000c00 tx=0x7f5298008030 comp rx=0 tx=0).stop 2023-11-24T06:08:15.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.082+0000 7f52aaa62700 1 -- 172.21.15.91:0/260198185 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f52a4161bc0 msgr2=0x7f52a40b1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:15.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.082+0000 7f52aaa62700 1 --2- 172.21.15.91:0/260198185 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f52a4161bc0 0x7f52a40b1670 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f529c00cbc0 tx=0x7f529c00ced0 comp rx=0 tx=0).stop 2023-11-24T06:08:15.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.082+0000 7f52aaa62700 1 -- 172.21.15.91:0/260198185 shutdown_connections 2023-11-24T06:08:15.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.082+0000 7f52aaa62700 1 --2- 172.21.15.91:0/260198185 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f529006aed0 0x7f529006d390 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:15.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.082+0000 7f52aaa62700 1 --2- 172.21.15.91:0/260198185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f52a406b6e0 0x7f52a4145e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:15.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.082+0000 7f52aaa62700 1 --2- 172.21.15.91:0/260198185 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f52a4161bc0 0x7f52a40b1670 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:15.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.082+0000 7f52aaa62700 1 --2- 172.21.15.91:0/260198185 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f52a415e230 0x7f52a406e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:15.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.082+0000 7f52aaa62700 1 -- 172.21.15.91:0/260198185 >> 172.21.15.91:0/260198185 conn(0x7f52a40d0a80 msgr2=0x7f52a40d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:15.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.082+0000 7f52aaa62700 1 -- 172.21.15.91:0/260198185 shutdown_connections 2023-11-24T06:08:15.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.082+0000 7f52aaa62700 1 -- 172.21.15.91:0/260198185 wait complete. 2023-11-24T06:08:15.099 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:08:15.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.215+0000 7f888aa07700 1 Processor -- start 2023-11-24T06:08:15.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.215+0000 7f888aa07700 1 -- start start 2023-11-24T06:08:15.225 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f888aa07700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f88840cf2b0 0x7f888415c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:15.225 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f888aa07700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88840d0a90 con 0x7f88840ce5c0 2023-11-24T06:08:15.225 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f888aa07700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888406ee10 con 0x7f888415bde0 2023-11-24T06:08:15.226 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f888aa07700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888406f0f0 con 0x7f88840cf2b0 2023-11-24T06:08:15.226 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f8888fa4700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f88840ce5c0 0x7f888406fbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53346/0 (socket says 172.21.15.91:53346) 2023-11-24T06:08:15.226 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f8888fa4700 1 -- 172.21.15.91:0/2271041076 learned_addr learned my addr 172.21.15.91:0/2271041076 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:15.226 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f88837fe700 1 --2- 172.21.15.91:0/2271041076 >> v2:172.21.15.94:3300/0 conn(0x7f88840cf2b0 0x7f888415c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:15.226 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f8882ffd700 1 -- 172.21.15.91:0/2271041076 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156558188 0 0) 0x7f88840d0a90 con 0x7f88840ce5c0 2023-11-24T06:08:15.226 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f8882ffd700 1 -- 172.21.15.91:0/2271041076 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f886c0035b0 con 0x7f88840ce5c0 2023-11-24T06:08:15.226 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f8882ffd700 1 -- 172.21.15.91:0/2271041076 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1737578170 0 0) 0x7f886c0035b0 con 0x7f88840ce5c0 2023-11-24T06:08:15.226 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f8882ffd700 1 -- 172.21.15.91:0/2271041076 >> v2:172.21.15.94:3300/0 conn(0x7f88840cf2b0 msgr2=0x7f888415c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:15.226 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f8882ffd700 1 --2- 172.21.15.91:0/2271041076 >> v2:172.21.15.94:3300/0 conn(0x7f88840cf2b0 0x7f888415c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:15.226 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f8882ffd700 1 -- 172.21.15.91:0/2271041076 >> v1:172.21.15.94:6789/0 conn(0x7f888415bde0 legacy=0x7f88840d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:15.226 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f8882ffd700 1 -- 172.21.15.91:0/2271041076 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888406f3d0 con 0x7f88840ce5c0 2023-11-24T06:08:15.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f8882ffd700 1 -- 172.21.15.91:0/2271041076 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f887c0019f0 con 0x7f88840ce5c0 2023-11-24T06:08:15.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f8882ffd700 1 -- 172.21.15.91:0/2271041076 >> v1:172.21.15.91:6789/0 conn(0x7f88840ce5c0 legacy=0x7f888406fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:15.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f8882ffd700 1 --2- 172.21.15.91:0/2271041076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f886c003a00 0x7f886c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:15.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f8882ffd700 1 --2- 172.21.15.91:0/2271041076 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f886c004850 0x7f886c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:15.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.216+0000 7f8882ffd700 1 --2- 172.21.15.91:0/2271041076 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f886c005210 0x7f886c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:15.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.217+0000 7f8882ffd700 1 -- 172.21.15.91:0/2271041076 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888406f0f0 con 0x7f886c005210 2023-11-24T06:08:15.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.217+0000 7f8882ffd700 1 -- 172.21.15.91:0/2271041076 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c00c1b0 con 0x7f886c003a00 2023-11-24T06:08:15.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.217+0000 7f8882ffd700 1 -- 172.21.15.91:0/2271041076 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886c00c370 con 0x7f886c004850 2023-11-24T06:08:15.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.217+0000 7f8888fa4700 1 --2- 172.21.15.91:0/2271041076 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f886c005210 0x7f886c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:15.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.217+0000 7f8883fff700 1 --2- 172.21.15.91:0/2271041076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f886c003a00 0x7f886c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:15.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.217+0000 7f88837fe700 1 --2- 172.21.15.91:0/2271041076 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f886c004850 0x7f886c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:15.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.217+0000 7f8888fa4700 1 -- 172.21.15.91:0/2271041076 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f886c004850 msgr2=0x7f886c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:15.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.217+0000 7f8888fa4700 1 --2- 172.21.15.91:0/2271041076 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f886c004850 0x7f886c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:15.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.217+0000 7f8888fa4700 1 -- 172.21.15.91:0/2271041076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f886c003a00 msgr2=0x7f886c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:15.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.217+0000 7f8888fa4700 1 --2- 172.21.15.91:0/2271041076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f886c003a00 0x7f886c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:15.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.217+0000 7f8888fa4700 1 -- 172.21.15.91:0/2271041076 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886c0035b0 con 0x7f886c005210 2023-11-24T06:08:15.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.217+0000 7f8888fa4700 1 -- 172.21.15.91:0/2271041076 reap_dead start 2023-11-24T06:08:15.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.217+0000 7f8888fa4700 1 --2- 172.21.15.91:0/2271041076 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f886c005210 0x7f886c00ba10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f887c00a830 tx=0x7f886c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0899f721479d75 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:15.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.217+0000 7f8882ffd700 1 -- 172.21.15.91:0/2271041076 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c00a2f0 con 0x7f886c005210 2023-11-24T06:08:15.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.217+0000 7f8882ffd700 1 -- 172.21.15.91:0/2271041076 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887c012600 con 0x7f886c005210 2023-11-24T06:08:15.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.217+0000 7f8882ffd700 1 -- 172.21.15.91:0/2271041076 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c0129b0 con 0x7f886c005210 2023-11-24T06:08:15.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.218+0000 7f888aa07700 1 -- 172.21.15.91:0/2271041076 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f886c005210 msgr2=0x7f886c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:15.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.218+0000 7f888aa07700 1 --2- 172.21.15.91:0/2271041076 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f886c005210 0x7f886c00ba10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f887c00a830 tx=0x7f886c005880 comp rx=0 tx=0).stop 2023-11-24T06:08:15.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.218+0000 7f888aa07700 1 -- 172.21.15.91:0/2271041076 shutdown_connections 2023-11-24T06:08:15.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.218+0000 7f888aa07700 1 --2- 172.21.15.91:0/2271041076 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f886c005210 0x7f886c00ba10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:15.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.218+0000 7f888aa07700 1 -- 172.21.15.91:0/2271041076 >> 172.21.15.91:0/2271041076 conn(0x7f88840b3720 msgr2=0x7f88840b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:15.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.218+0000 7f888aa07700 1 -- 172.21.15.91:0/2271041076 shutdown_connections 2023-11-24T06:08:15.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.218+0000 7f888aa07700 1 -- 172.21.15.91:0/2271041076 wait complete. 2023-11-24T06:08:15.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.219+0000 7f888aa07700 1 Processor -- start 2023-11-24T06:08:15.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.219+0000 7f888aa07700 1 -- start start 2023-11-24T06:08:15.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f888aa07700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f886c005210 0x7f88840c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:15.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f888aa07700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88841591c0 0x7f88841f34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:15.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f888aa07700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f88840c96b0 0x7f88840c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:15.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f888aa07700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88840ca0a0 con 0x7f886c005210 2023-11-24T06:08:15.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f888aa07700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88841f3a10 con 0x7f88841591c0 2023-11-24T06:08:15.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f888aa07700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8884159680 con 0x7f88840c96b0 2023-11-24T06:08:15.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f8883fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f886c005210 0x7f88840c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:15.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f8883fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f886c005210 0x7f88840c90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58420/0 (socket says 172.21.15.91:58420) 2023-11-24T06:08:15.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f8883fff700 1 -- 172.21.15.91:0/940435278 learned_addr learned my addr 172.21.15.91:0/940435278 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:15.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f88837fe700 1 --2- 172.21.15.91:0/940435278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88841591c0 0x7f88841f34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:15.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f8888fa4700 1 --2- 172.21.15.91:0/940435278 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f88840c96b0 0x7f88840c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:15.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f8883fff700 1 -- 172.21.15.91:0/940435278 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f88840c96b0 msgr2=0x7f88840c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:15.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f8883fff700 1 --2- 172.21.15.91:0/940435278 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f88840c96b0 0x7f88840c9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:15.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f8883fff700 1 -- 172.21.15.91:0/940435278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88841591c0 msgr2=0x7f88841f34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:15.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f8883fff700 1 --2- 172.21.15.91:0/940435278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88841591c0 0x7f88841f34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:15.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f8883fff700 1 -- 172.21.15.91:0/940435278 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8884159960 con 0x7f886c005210 2023-11-24T06:08:15.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f8888fa4700 1 --2- 172.21.15.91:0/940435278 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f88840c96b0 0x7f88840c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:15.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f8883fff700 1 --2- 172.21.15.91:0/940435278 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f886c005210 0x7f88840c90b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f887800f7f0 tx=0x7f887800fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:15.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f8880ff9700 1 -- 172.21.15.91:0/940435278 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88780052e0 con 0x7f886c005210 2023-11-24T06:08:15.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f8880ff9700 1 -- 172.21.15.91:0/940435278 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8878005440 con 0x7f886c005210 2023-11-24T06:08:15.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f888aa07700 1 -- 172.21.15.91:0/940435278 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8884159c40 con 0x7f886c005210 2023-11-24T06:08:15.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f8880ff9700 1 -- 172.21.15.91:0/940435278 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88780038a0 con 0x7f886c005210 2023-11-24T06:08:15.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.220+0000 7f888aa07700 1 -- 172.21.15.91:0/940435278 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888415a200 con 0x7f886c005210 2023-11-24T06:08:15.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.221+0000 7f888aa07700 1 -- 172.21.15.91:0/940435278 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8884068670 con 0x7f886c005210 2023-11-24T06:08:15.232 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.222+0000 7f8880ff9700 1 -- 172.21.15.91:0/940435278 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8878003a00 con 0x7f886c005210 2023-11-24T06:08:15.232 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.222+0000 7f8880ff9700 1 --2- 172.21.15.91:0/940435278 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f887006aed0 0x7f887006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:15.232 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.222+0000 7f8880ff9700 1 -- 172.21.15.91:0/940435278 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8878089380 con 0x7f886c005210 2023-11-24T06:08:15.232 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.222+0000 7f88837fe700 1 --2- 172.21.15.91:0/940435278 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f887006aed0 0x7f887006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:15.232 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.223+0000 7f88837fe700 1 --2- 172.21.15.91:0/940435278 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f887006aed0 0x7f887006d390 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f88840cf710 tx=0x7f8874009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:15.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.226+0000 7f8880ff9700 1 -- 172.21.15.91:0/940435278 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8878054440 con 0x7f886c005210 2023-11-24T06:08:15.464 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.460+0000 7f888aa07700 1 -- 172.21.15.91:0/940435278 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88840702a0 con 0x7f886c005210 2023-11-24T06:08:15.464 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.460+0000 7f8880ff9700 1 -- 172.21.15.91:0/940435278 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4907 (secure 0 0 0) 0x7f8878003cb0 con 0x7f886c005210 2023-11-24T06:08:15.465 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:15.465 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1329219967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1329219967},{"type":"v1","addr":"172.21.15.94:6834","nonce":1329219967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3439524864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3439524864},{"type":"v1","addr":"172.21.15.94:6835","nonce":3439524864}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6836/3390368219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6834","nonce":3390368219},{"type":"v1","addr":"172.21.15.91:6836","nonce":3390368219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:08:00.798681+0000","modified":"2023-11-24T06:08:04.132764+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.91:6837/1644079645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":1644079645},{"type":"v1","addr":"172.21.15.91:6837","nonce":1644079645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T06:08:15.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.462+0000 7f888aa07700 1 -- 172.21.15.91:0/940435278 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f887006aed0 msgr2=0x7f887006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:15.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.462+0000 7f888aa07700 1 --2- 172.21.15.91:0/940435278 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f887006aed0 0x7f887006d390 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f88840cf710 tx=0x7f8874009210 comp rx=0 tx=0).stop 2023-11-24T06:08:15.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.462+0000 7f888aa07700 1 -- 172.21.15.91:0/940435278 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f886c005210 msgr2=0x7f88840c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:15.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.462+0000 7f888aa07700 1 --2- 172.21.15.91:0/940435278 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f886c005210 0x7f88840c90b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f887800f7f0 tx=0x7f887800fbb0 comp rx=0 tx=0).stop 2023-11-24T06:08:15.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.463+0000 7f888aa07700 1 -- 172.21.15.91:0/940435278 shutdown_connections 2023-11-24T06:08:15.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.463+0000 7f888aa07700 1 --2- 172.21.15.91:0/940435278 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f887006aed0 0x7f887006d390 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:15.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.463+0000 7f888aa07700 1 --2- 172.21.15.91:0/940435278 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f88840c96b0 0x7f88840c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:15.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.463+0000 7f888aa07700 1 --2- 172.21.15.91:0/940435278 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f886c005210 0x7f88840c90b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:15.467 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.463+0000 7f888aa07700 1 --2- 172.21.15.91:0/940435278 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f88841591c0 0x7f88841f34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:15.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.463+0000 7f888aa07700 1 -- 172.21.15.91:0/940435278 >> 172.21.15.91:0/940435278 conn(0x7f88840b3720 msgr2=0x7f88840707c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:15.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.463+0000 7f888aa07700 1 -- 172.21.15.91:0/940435278 shutdown_connections 2023-11-24T06:08:15.468 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:15.463+0000 7f888aa07700 1 -- 172.21.15.91:0/940435278 wait complete. 2023-11-24T06:08:15.468 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 8 2023-11-24T06:08:15.881 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T06:08:00.798681+0000', 'modified': '2023-11-24T06:08:04.132764+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4290': {'gid': 4290, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.91:6837/1644079645', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.91:6835', 'nonce': 1644079645}, {'type': 'v1', 'addr': '172.21.15.91:6837', 'nonce': 1644079645}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T06:08:15.881 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T06:08:15.881 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-11-24T06:08:16.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.000+0000 7fc85f501700 1 Processor -- start 2023-11-24T06:08:16.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.000+0000 7fc85f501700 1 -- start start 2023-11-24T06:08:16.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.001+0000 7fc85f501700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fc85815a440 0x7fc8580cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.001+0000 7fc85f501700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8580b05f0 con 0x7fc85815a820 2023-11-24T06:08:16.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.001+0000 7fc85f501700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8580734c0 con 0x7fc85815eb80 2023-11-24T06:08:16.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.001+0000 7fc85f501700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc85815f080 con 0x7fc85815a440 2023-11-24T06:08:16.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.001+0000 7fc85d29d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fc85815a440 0x7fc8580cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.001+0000 7fc85ca9c700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fc85815a820 0x7fc8580726a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53364/0 (socket says 172.21.15.91:53364) 2023-11-24T06:08:16.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.001+0000 7fc85ca9c700 1 -- 172.21.15.91:0/631740152 learned_addr learned my addr 172.21.15.91:0/631740152 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:16.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.001+0000 7fc85d29d700 1 -- 172.21.15.91:0/631740152 >> v1:172.21.15.94:6789/0 conn(0x7fc85815eb80 legacy=0x7fc858072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:16.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.001+0000 7fc85d29d700 1 -- 172.21.15.91:0/631740152 >> v1:172.21.15.91:6789/0 conn(0x7fc85815a820 legacy=0x7fc8580726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:16.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.001+0000 7fc85d29d700 1 -- 172.21.15.91:0/631740152 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8580afd50 con 0x7fc85815a440 2023-11-24T06:08:16.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.001+0000 7fc84ffff700 1 -- 172.21.15.91:0/631740152 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096480147 0 0) 0x7fc8580b05f0 con 0x7fc85815a820 2023-11-24T06:08:16.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.001+0000 7fc85d29d700 1 --2- 172.21.15.91:0/631740152 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc85815a440 0x7fc8580cffa0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc854009840 tx=0x7fc854009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3c54ee74ea08c9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:16.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.001+0000 7fc84ffff700 1 -- 172.21.15.91:0/631740152 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc854004230 con 0x7fc85815a440 2023-11-24T06:08:16.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.001+0000 7fc84ffff700 1 -- 172.21.15.91:0/631740152 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc854004390 con 0x7fc85815a440 2023-11-24T06:08:16.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.001+0000 7fc84ffff700 1 -- 172.21.15.91:0/631740152 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8540036a0 con 0x7fc85815a440 2023-11-24T06:08:16.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.002+0000 7fc85f501700 1 -- 172.21.15.91:0/631740152 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc85815a440 msgr2=0x7fc8580cffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:16.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.002+0000 7fc85f501700 1 --2- 172.21.15.91:0/631740152 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc85815a440 0x7fc8580cffa0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc854009840 tx=0x7fc854009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:16.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.002+0000 7fc85f501700 1 -- 172.21.15.91:0/631740152 shutdown_connections 2023-11-24T06:08:16.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.002+0000 7fc85f501700 1 --2- 172.21.15.91:0/631740152 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc85815a440 0x7fc8580cffa0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.002+0000 7fc85f501700 1 -- 172.21.15.91:0/631740152 >> 172.21.15.91:0/631740152 conn(0x7fc8580b5070 msgr2=0x7fc8580b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:16.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.002+0000 7fc85f501700 1 -- 172.21.15.91:0/631740152 shutdown_connections 2023-11-24T06:08:16.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.002+0000 7fc85f501700 1 -- 172.21.15.91:0/631740152 wait complete. 2023-11-24T06:08:16.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.003+0000 7fc85f501700 1 Processor -- start 2023-11-24T06:08:16.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.003+0000 7fc85f501700 1 -- start start 2023-11-24T06:08:16.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85f501700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc85815a440 0x7fc8580d0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85f501700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc85815eb80 0x7fc8580cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85f501700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc8580ce280 0x7fc85814a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85f501700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8580ce790 con 0x7fc8580ce280 2023-11-24T06:08:16.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85f501700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8580cea70 con 0x7fc85815a440 2023-11-24T06:08:16.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85f501700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8580ced50 con 0x7fc85815eb80 2023-11-24T06:08:16.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85da9e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc8580ce280 0x7fc85814a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85da9e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc8580ce280 0x7fc85814a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58438/0 (socket says 172.21.15.91:58438) 2023-11-24T06:08:16.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85da9e700 1 -- 172.21.15.91:0/1537283555 learned_addr learned my addr 172.21.15.91:0/1537283555 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:16.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85d29d700 1 --2- 172.21.15.91:0/1537283555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc85815a440 0x7fc8580d0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85ca9c700 1 --2- 172.21.15.91:0/1537283555 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc85815eb80 0x7fc8580cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85da9e700 1 -- 172.21.15.91:0/1537283555 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc85815eb80 msgr2=0x7fc8580cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:16.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85da9e700 1 --2- 172.21.15.91:0/1537283555 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc85815eb80 0x7fc8580cdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85da9e700 1 -- 172.21.15.91:0/1537283555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc85815a440 msgr2=0x7fc8580d0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:16.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85da9e700 1 --2- 172.21.15.91:0/1537283555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc85815a440 0x7fc8580d0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85da9e700 1 -- 172.21.15.91:0/1537283555 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8540094a0 con 0x7fc8580ce280 2023-11-24T06:08:16.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85ca9c700 1 --2- 172.21.15.91:0/1537283555 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc85815eb80 0x7fc8580cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:16.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85d29d700 1 --2- 172.21.15.91:0/1537283555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc85815a440 0x7fc8580d0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:16.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.004+0000 7fc85da9e700 1 --2- 172.21.15.91:0/1537283555 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc8580ce280 0x7fc85814a280 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc85000cba0 tx=0x7fc85000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:16.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.005+0000 7fc84dffb700 1 -- 172.21.15.91:0/1537283555 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85000ab80 con 0x7fc8580ce280 2023-11-24T06:08:16.012 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.005+0000 7fc84dffb700 1 -- 172.21.15.91:0/1537283555 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85000ace0 con 0x7fc8580ce280 2023-11-24T06:08:16.012 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.005+0000 7fc85f501700 1 -- 172.21.15.91:0/1537283555 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8580cf030 con 0x7fc8580ce280 2023-11-24T06:08:16.012 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.005+0000 7fc84dffb700 1 -- 172.21.15.91:0/1537283555 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc850012660 con 0x7fc8580ce280 2023-11-24T06:08:16.012 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.005+0000 7fc85f501700 1 -- 172.21.15.91:0/1537283555 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8580cf4e0 con 0x7fc8580ce280 2023-11-24T06:08:16.012 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.006+0000 7fc84dffb700 1 -- 172.21.15.91:0/1537283555 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc8500127c0 con 0x7fc8580ce280 2023-11-24T06:08:16.013 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.007+0000 7fc84dffb700 1 --2- 172.21.15.91:0/1537283555 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc83c06aed0 0x7fc83c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.013 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.007+0000 7fc84dffb700 1 -- 172.21.15.91:0/1537283555 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc850089e40 con 0x7fc8580ce280 2023-11-24T06:08:16.013 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.007+0000 7fc85d29d700 1 --2- 172.21.15.91:0/1537283555 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc83c06aed0 0x7fc83c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.013 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.007+0000 7fc85d29d700 1 --2- 172.21.15.91:0/1537283555 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc83c06aed0 0x7fc83c06d390 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc8540090b0 tx=0x7fc854008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:16.013 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.009+0000 7fc86b6fab80 1 Processor -- start 2023-11-24T06:08:16.013 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.009+0000 7fc86b6fab80 1 -- start start 2023-11-24T06:08:16.015 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc86b6fab80 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df432b080 0x558df4371840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.015 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc86b6fab80 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4340ed0 0x558df4371d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc86b6fab80 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df4328490 0x558df4354e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc86b6fab80 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df429e770 con 0x558df4340ed0 2023-11-24T06:08:16.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc86b6fab80 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df4285750 con 0x558df4328490 2023-11-24T06:08:16.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc86b6fab80 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x558df4283070 con 0x558df432b080 2023-11-24T06:08:16.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc85ca9c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4340ed0 0x558df4371d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc85ca9c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4340ed0 0x558df4371d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58444/0 (socket says 172.21.15.91:58444) 2023-11-24T06:08:16.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc85ca9c700 1 -- 172.21.15.91:0/2783720951 learned_addr learned my addr 172.21.15.91:0/2783720951 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:16.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc85d29d700 1 --2- 172.21.15.91:0/2783720951 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df432b080 0x558df4371840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc85da9e700 1 --2- 172.21.15.91:0/2783720951 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df4328490 0x558df4354e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc85ca9c700 1 -- 172.21.15.91:0/2783720951 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df432b080 msgr2=0x558df4371840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:16.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc85ca9c700 1 --2- 172.21.15.91:0/2783720951 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df432b080 0x558df4371840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc85ca9c700 1 -- 172.21.15.91:0/2783720951 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df4328490 msgr2=0x558df4354e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:16.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc85ca9c700 1 --2- 172.21.15.91:0/2783720951 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df4328490 0x558df4354e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc85ca9c700 1 -- 172.21.15.91:0/2783720951 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558df40fb5e0 con 0x558df4340ed0 2023-11-24T06:08:16.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc85ca9c700 1 --2- 172.21.15.91:0/2783720951 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4340ed0 0x558df4371d80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc848001840 tx=0x7fc848001b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79abf3c729755338 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:16.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc842ffd700 1 -- 172.21.15.91:0/2783720951 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc848004c80 con 0x558df4340ed0 2023-11-24T06:08:16.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc842ffd700 1 -- 172.21.15.91:0/2783720951 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc848004de0 con 0x558df4340ed0 2023-11-24T06:08:16.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.010+0000 7fc842ffd700 1 -- 172.21.15.91:0/2783720951 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8480072f0 con 0x558df4340ed0 2023-11-24T06:08:16.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.011+0000 7fc86b6fab80 1 -- 172.21.15.91:0/2783720951 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4340ed0 msgr2=0x558df4371d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:16.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.011+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/2783720951 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4340ed0 0x558df4371d80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc848001840 tx=0x7fc848001b50 comp rx=0 tx=0).stop 2023-11-24T06:08:16.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.011+0000 7fc86b6fab80 1 -- 172.21.15.91:0/2783720951 shutdown_connections 2023-11-24T06:08:16.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.011+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/2783720951 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df4328490 0x558df4354e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.011+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/2783720951 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df432b080 0x558df4371840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.011+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/2783720951 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4340ed0 0x558df4371d80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.011+0000 7fc86b6fab80 1 -- 172.21.15.91:0/2783720951 >> 172.21.15.91:0/2783720951 conn(0x558df4259310 msgr2=0x558df436fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:16.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.011+0000 7fc86b6fab80 1 -- 172.21.15.91:0/2783720951 shutdown_connections 2023-11-24T06:08:16.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.011+0000 7fc86b6fab80 1 -- 172.21.15.91:0/2783720951 wait complete. 2023-11-24T06:08:16.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.012+0000 7fc86b6fab80 1 -- start start 2023-11-24T06:08:16.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.012+0000 7fc86b6fab80 1 Processor -- start 2023-11-24T06:08:16.021 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.012+0000 7fc86b6fab80 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4328490 0x558df437ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.021 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc86b6fab80 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df432b080 0x558df436b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc86b6fab80 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df4340ed0 0x558df436b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc86b6fab80 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df427e960 con 0x558df4328490 2023-11-24T06:08:16.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc86b6fab80 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df4238510 con 0x558df432b080 2023-11-24T06:08:16.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc86b6fab80 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x558df42381b0 con 0x558df4340ed0 2023-11-24T06:08:16.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc85d29d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4328490 0x558df437ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc86b6fab80 10 client.0 _ll_register_callbacks cb 0x558df4163a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T06:08:16.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc85d29d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4328490 0x558df437ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58448/0 (socket says 172.21.15.91:58448) 2023-11-24T06:08:16.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc85d29d700 1 -- 172.21.15.91:0/3752382944 learned_addr learned my addr 172.21.15.91:0/3752382944 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:16.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc85da9e700 1 --2- 172.21.15.91:0/3752382944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df4340ed0 0x558df436b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc85ca9c700 1 --2- 172.21.15.91:0/3752382944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df432b080 0x558df436b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc85d29d700 1 -- 172.21.15.91:0/3752382944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df4340ed0 msgr2=0x558df436b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:16.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc85d29d700 1 --2- 172.21.15.91:0/3752382944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df4340ed0 0x558df436b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc85d29d700 1 -- 172.21.15.91:0/3752382944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df432b080 msgr2=0x558df436b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:16.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc85d29d700 1 --2- 172.21.15.91:0/3752382944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df432b080 0x558df436b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc85d29d700 1 -- 172.21.15.91:0/3752382944 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8480014a0 con 0x558df4328490 2023-11-24T06:08:16.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc85d29d700 1 --2- 172.21.15.91:0/3752382944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4328490 0x558df437ef40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc8580737f0 tx=0x7fc8540047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d16326d443a43f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:16.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc837fff700 10 client.0 ms_handle_connect on v2:172.21.15.91:3300/0 2023-11-24T06:08:16.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc837fff700 1 -- 172.21.15.91:0/3752382944 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc854011a40 con 0x558df4328490 2023-11-24T06:08:16.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc837fff700 1 -- 172.21.15.91:0/3752382944 <== mon.0 v2:172.21.15.91:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc854011ba0 con 0x558df4328490 2023-11-24T06:08:16.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc86b6fab80 1 -- 172.21.15.91:0/3752382944 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558df431da40 con 0x558df4328490 2023-11-24T06:08:16.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc837fff700 1 -- 172.21.15.91:0/3752382944 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc854011d60 con 0x558df4328490 2023-11-24T06:08:16.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc86b6fab80 10 client.4653 fetch_fsmap learned FSMap version 8 2023-11-24T06:08:16.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc86b6fab80 1 -- 172.21.15.91:0/3752382944 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x558df40a7d90 con 0x558df4328490 2023-11-24T06:08:16.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.013+0000 7fc837fff700 1 -- 172.21.15.91:0/3752382944 <== mon.0 v2:172.21.15.91:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558df431da40 con 0x558df4328490 2023-11-24T06:08:16.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.014+0000 7fc837fff700 1 -- 172.21.15.91:0/3752382944 <== mon.0 v2:172.21.15.91:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc854031440 con 0x558df4328490 2023-11-24T06:08:16.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.014+0000 7fc86b6fab80 10 client.4653 fetch_fsmap finished waiting for FSMap version 8 2023-11-24T06:08:16.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.014+0000 7fc86b6fab80 10 client.4653 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-24T06:08:16.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.014+0000 7fc86b6fab80 20 client.4653 populate_metadata read hostname 'smithi091' 2023-11-24T06:08:16.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.014+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/3752382944 >> [v2:172.21.15.91:6835/1644079645,v1:172.21.15.91:6837/1644079645] conn(0x558df434e7d0 0x558df438ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.014+0000 7fc86b6fab80 4 client.4653 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T06:08:16.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.014+0000 7fc86b6fab80 1 -- 172.21.15.91:0/3752382944 --> [v2:172.21.15.91:6835/1644079645,v1:172.21.15.91:6837/1644079645] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558df42f9840 con 0x558df434e7d0 2023-11-24T06:08:16.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.014+0000 7fc85ca9c700 1 --2- 172.21.15.91:0/3752382944 >> [v2:172.21.15.91:6835/1644079645,v1:172.21.15.91:6837/1644079645] conn(0x558df434e7d0 0x558df438ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.014+0000 7fc837fff700 10 client.4653 ms_handle_connect on v2:172.21.15.91:6835/1644079645 2023-11-24T06:08:16.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.014+0000 7fc85ca9c700 1 --2- 172.21.15.91:0/3752382944 >> [v2:172.21.15.91:6835/1644079645,v1:172.21.15.91:6837/1644079645] conn(0x558df434e7d0 0x558df438ebb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46793b57686a3cae server_cookie=83d451b0412effa4 in_seq=0 out_seq=0 2023-11-24T06:08:16.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.015+0000 7fc837fff700 1 -- 172.21.15.91:0/3752382944 <== mds.0 v2:172.21.15.91:6835/1644079645 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558df42f9840 con 0x558df434e7d0 2023-11-24T06:08:16.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.015+0000 7fc837fff700 10 client.4653 handle_command_reply: tid=0 2023-11-24T06:08:16.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.015+0000 7fc86b6fab80 1 client.4653 shutdown 2023-11-24T06:08:16.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.015+0000 7fc86b6fab80 1 -- 172.21.15.91:0/3752382944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4328490 msgr2=0x558df437ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:16.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.015+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/3752382944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4328490 0x558df437ef40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc8580737f0 tx=0x7fc8540047c0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.015+0000 7fc86b6fab80 1 -- 172.21.15.91:0/3752382944 shutdown_connections 2023-11-24T06:08:16.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.015+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/3752382944 >> [v2:172.21.15.91:6835/1644079645,v1:172.21.15.91:6837/1644079645] conn(0x558df434e7d0 0x558df438ebb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.015+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/3752382944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df4340ed0 0x558df436b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.015+0000 7fc837fff700 0 client.4653 ms_handle_reset on v2:172.21.15.91:6835/1644079645 2023-11-24T06:08:16.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.015+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/3752382944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4328490 0x558df437ef40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.015+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/3752382944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df432b080 0x558df436b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.015+0000 7fc86b6fab80 1 -- 172.21.15.91:0/3752382944 >> 172.21.15.91:0/3752382944 conn(0x558df4259310 msgr2=0x558df4373c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:16.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.015+0000 7fc86b6fab80 1 -- 172.21.15.91:0/3752382944 shutdown_connections 2023-11-24T06:08:16.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.015+0000 7fc86b6fab80 1 -- 172.21.15.91:0/3752382944 wait complete. 2023-11-24T06:08:16.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.015+0000 7fc86b6fab80 20 client.4653 trim_cache size 0 max 16384 2023-11-24T06:08:16.037 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.033+0000 7fc86b6fab80 1 Processor -- start 2023-11-24T06:08:16.037 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.033+0000 7fc86b6fab80 1 -- start start 2023-11-24T06:08:16.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc86b6fab80 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df4328490 0x558df4371df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc86b6fab80 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df432b080 0x558df4351370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc86b6fab80 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df4340ed0 0x558df4325230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc86b6fab80 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df436da00 con 0x558df432b080 2023-11-24T06:08:16.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc86b6fab80 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df40fd8d0 con 0x558df4328490 2023-11-24T06:08:16.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc86b6fab80 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x558df4393a40 con 0x558df4340ed0 2023-11-24T06:08:16.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc85ca9c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df432b080 0x558df4351370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc85ca9c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df432b080 0x558df4351370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58458/0 (socket says 172.21.15.91:58458) 2023-11-24T06:08:16.040 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc85ca9c700 1 -- 172.21.15.91:0/2385918916 learned_addr learned my addr 172.21.15.91:0/2385918916 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:16.040 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc85ca9c700 1 -- 172.21.15.91:0/2385918916 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df4340ed0 msgr2=0x558df4325230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:16.040 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc85ca9c700 1 --2- 172.21.15.91:0/2385918916 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df4340ed0 0x558df4325230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.040 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc85ca9c700 1 -- 172.21.15.91:0/2385918916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df4328490 msgr2=0x558df4371df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:16.040 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc85ca9c700 1 --2- 172.21.15.91:0/2385918916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df4328490 0x558df4371df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.040 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc85ca9c700 1 -- 172.21.15.91:0/2385918916 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558df4090c10 con 0x558df432b080 2023-11-24T06:08:16.040 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc85ca9c700 1 --2- 172.21.15.91:0/2385918916 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df432b080 0x558df4351370 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc848001790 tx=0x7fc848006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=391942c9ee7b6a42 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:16.040 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc8437fe700 1 -- 172.21.15.91:0/2385918916 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc848026070 con 0x558df432b080 2023-11-24T06:08:16.040 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc8437fe700 1 -- 172.21.15.91:0/2385918916 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc848004a80 con 0x558df432b080 2023-11-24T06:08:16.040 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.034+0000 7fc8437fe700 1 -- 172.21.15.91:0/2385918916 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc848004c50 con 0x558df432b080 2023-11-24T06:08:16.040 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.035+0000 7fc86b6fab80 1 -- 172.21.15.91:0/2385918916 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df432b080 msgr2=0x558df4351370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:16.040 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.035+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/2385918916 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df432b080 0x558df4351370 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc848001790 tx=0x7fc848006000 comp rx=0 tx=0).stop 2023-11-24T06:08:16.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.035+0000 7fc86b6fab80 1 -- 172.21.15.91:0/2385918916 shutdown_connections 2023-11-24T06:08:16.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.035+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/2385918916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df4328490 0x558df4371df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.035+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/2385918916 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df4340ed0 0x558df4325230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.035+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/2385918916 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df432b080 0x558df4351370 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.035+0000 7fc86b6fab80 1 -- 172.21.15.91:0/2385918916 >> 172.21.15.91:0/2385918916 conn(0x558df4259310 msgr2=0x558df436f010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:16.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.035+0000 7fc86b6fab80 1 -- 172.21.15.91:0/2385918916 shutdown_connections 2023-11-24T06:08:16.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.035+0000 7fc86b6fab80 1 -- 172.21.15.91:0/2385918916 wait complete. 2023-11-24T06:08:16.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.035+0000 7fc86b6fab80 1 -- start start 2023-11-24T06:08:16.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.035+0000 7fc86b6fab80 1 Processor -- start 2023-11-24T06:08:16.044 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc86b6fab80 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df4328490 0x558df438f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.045 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc86b6fab80 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df432b080 0x558df43511c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.045 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc86b6fab80 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4340ed0 0x558df4351700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.045 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc86b6fab80 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df438d990 con 0x558df4340ed0 2023-11-24T06:08:16.045 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc86b6fab80 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x558df434f240 con 0x558df432b080 2023-11-24T06:08:16.045 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc86b6fab80 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x558df438dbe0 con 0x558df4328490 2023-11-24T06:08:16.045 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc86b6fab80 10 client.0 _ll_register_callbacks cb 0x558df4163a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T06:08:16.045 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc85da9e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4340ed0 0x558df4351700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.045 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc85da9e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4340ed0 0x558df4351700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58466/0 (socket says 172.21.15.91:58466) 2023-11-24T06:08:16.045 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc85da9e700 1 -- 172.21.15.91:0/1453695920 learned_addr learned my addr 172.21.15.91:0/1453695920 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:16.045 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc85d29d700 1 --2- 172.21.15.91:0/1453695920 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df4328490 0x558df438f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.045 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc85ca9c700 1 --2- 172.21.15.91:0/1453695920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df432b080 0x558df43511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.045 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc85da9e700 1 -- 172.21.15.91:0/1453695920 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df4328490 msgr2=0x558df438f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:16.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc85da9e700 1 --2- 172.21.15.91:0/1453695920 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df4328490 0x558df438f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc85da9e700 1 -- 172.21.15.91:0/1453695920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df432b080 msgr2=0x558df43511c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:16.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc85da9e700 1 --2- 172.21.15.91:0/1453695920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df432b080 0x558df43511c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc85da9e700 1 -- 172.21.15.91:0/1453695920 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558df40fb5e0 con 0x558df4340ed0 2023-11-24T06:08:16.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc85da9e700 1 --2- 172.21.15.91:0/1453695920 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4340ed0 0x558df4351700 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc8500550b0 tx=0x7fc850004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc0262e391a855 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:16.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc837fff700 10 client.0 ms_handle_connect on v2:172.21.15.91:3300/0 2023-11-24T06:08:16.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc837fff700 1 -- 172.21.15.91:0/1453695920 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8500582a0 con 0x558df4340ed0 2023-11-24T06:08:16.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc837fff700 1 -- 172.21.15.91:0/1453695920 <== mon.0 v2:172.21.15.91:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc85005cd00 con 0x558df4340ed0 2023-11-24T06:08:16.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.037+0000 7fc86b6fab80 1 -- 172.21.15.91:0/1453695920 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc848004de0 con 0x558df4340ed0 2023-11-24T06:08:16.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc837fff700 1 -- 172.21.15.91:0/1453695920 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc850088120 con 0x558df4340ed0 2023-11-24T06:08:16.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc86b6fab80 10 client.4659 fetch_fsmap learned FSMap version 8 2023-11-24T06:08:16.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc86b6fab80 1 -- 172.21.15.91:0/1453695920 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x558df40ce200 con 0x558df4340ed0 2023-11-24T06:08:16.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc837fff700 1 -- 172.21.15.91:0/1453695920 <== mon.0 v2:172.21.15.91:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc848004de0 con 0x558df4340ed0 2023-11-24T06:08:16.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc837fff700 1 -- 172.21.15.91:0/1453695920 <== mon.0 v2:172.21.15.91:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc850077300 con 0x558df4340ed0 2023-11-24T06:08:16.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc86b6fab80 10 client.4659 fetch_fsmap finished waiting for FSMap version 8 2023-11-24T06:08:16.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc86b6fab80 10 client.4659 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-24T06:08:16.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc86b6fab80 20 client.4659 populate_metadata read hostname 'smithi091' 2023-11-24T06:08:16.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/1453695920 >> [v2:172.21.15.91:6835/1644079645,v1:172.21.15.91:6837/1644079645] conn(0x558df43db550 0x558df4370a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc86b6fab80 4 client.4659 mds_command: new command op to 4290 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T06:08:16.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc86b6fab80 1 -- 172.21.15.91:0/1453695920 --> [v2:172.21.15.91:6835/1644079645,v1:172.21.15.91:6837/1644079645] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558df4244f80 con 0x558df43db550 2023-11-24T06:08:16.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc85d29d700 1 --2- 172.21.15.91:0/1453695920 >> [v2:172.21.15.91:6835/1644079645,v1:172.21.15.91:6837/1644079645] conn(0x558df43db550 0x558df4370a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc837fff700 10 client.4659 ms_handle_connect on v2:172.21.15.91:6835/1644079645 2023-11-24T06:08:16.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc85d29d700 1 --2- 172.21.15.91:0/1453695920 >> [v2:172.21.15.91:6835/1644079645,v1:172.21.15.91:6837/1644079645] conn(0x558df43db550 0x558df4370a40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0f38d5ac7c03699 server_cookie=cac3fffce8ea77 in_seq=0 out_seq=0 2023-11-24T06:08:16.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc837fff700 1 -- 172.21.15.91:0/1453695920 <== mds.0 v2:172.21.15.91:6835/1644079645 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x558df4244f80 con 0x558df43db550 2023-11-24T06:08:16.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc837fff700 10 client.4659 handle_command_reply: tid=0 2023-11-24T06:08:16.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.038+0000 7fc86b6fab80 1 client.4659 shutdown 2023-11-24T06:08:16.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc86b6fab80 1 -- 172.21.15.91:0/1453695920 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4340ed0 msgr2=0x558df4351700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:16.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/1453695920 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4340ed0 0x558df4351700 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc8500550b0 tx=0x7fc850004360 comp rx=0 tx=0).stop 2023-11-24T06:08:16.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc86b6fab80 1 -- 172.21.15.91:0/1453695920 shutdown_connections 2023-11-24T06:08:16.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/1453695920 >> [v2:172.21.15.91:6835/1644079645,v1:172.21.15.91:6837/1644079645] conn(0x558df43db550 0x558df4370a40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/1453695920 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x558df4340ed0 0x558df4351700 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc837fff700 0 client.4659 ms_handle_reset on v2:172.21.15.91:6835/1644079645 2023-11-24T06:08:16.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/1453695920 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x558df4328490 0x558df438f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc86b6fab80 1 --2- 172.21.15.91:0/1453695920 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x558df432b080 0x558df43511c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc86b6fab80 1 -- 172.21.15.91:0/1453695920 >> 172.21.15.91:0/1453695920 conn(0x558df4259310 msgr2=0x558df43729a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:16.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc86b6fab80 1 -- 172.21.15.91:0/1453695920 shutdown_connections 2023-11-24T06:08:16.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc86b6fab80 1 -- 172.21.15.91:0/1453695920 wait complete. 2023-11-24T06:08:16.049 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc86b6fab80 20 client.4659 trim_cache size 0 max 16384 2023-11-24T06:08:16.049 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2023-11-24T06:08:16.049 INFO:teuthology.orchestra.run.smithi091.stdout: "cluster_fsid": "f7c07073-6cd6-496f-812a-c57c86a2fbbb", 2023-11-24T06:08:16.049 INFO:teuthology.orchestra.run.smithi091.stdout: "whoami": 0, 2023-11-24T06:08:16.049 INFO:teuthology.orchestra.run.smithi091.stdout: "id": 4290, 2023-11-24T06:08:16.049 INFO:teuthology.orchestra.run.smithi091.stdout: "want_state": "up:active", 2023-11-24T06:08:16.049 INFO:teuthology.orchestra.run.smithi091.stdout: "state": "up:active", 2023-11-24T06:08:16.049 INFO:teuthology.orchestra.run.smithi091.stdout: "fs_name": "cephfs", 2023-11-24T06:08:16.049 INFO:teuthology.orchestra.run.smithi091.stdout: "rank_uptime": 14.312040207000001, 2023-11-24T06:08:16.050 INFO:teuthology.orchestra.run.smithi091.stdout: "mdsmap_epoch": 8, 2023-11-24T06:08:16.050 INFO:teuthology.orchestra.run.smithi091.stdout: "osdmap_epoch": 20, 2023-11-24T06:08:16.050 INFO:teuthology.orchestra.run.smithi091.stdout: "osdmap_epoch_barrier": 20, 2023-11-24T06:08:16.050 INFO:teuthology.orchestra.run.smithi091.stdout: "uptime": 22.071221046000002 2023-11-24T06:08:16.050 INFO:teuthology.orchestra.run.smithi091.stdout:} 2023-11-24T06:08:16.051 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc85f501700 1 -- 172.21.15.91:0/1537283555 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc83c06aed0 msgr2=0x7fc83c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:16.051 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc85f501700 1 --2- 172.21.15.91:0/1537283555 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc83c06aed0 0x7fc83c06d390 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc8540090b0 tx=0x7fc854008030 comp rx=0 tx=0).stop 2023-11-24T06:08:16.051 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc85f501700 1 -- 172.21.15.91:0/1537283555 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc8580ce280 msgr2=0x7fc85814a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:16.051 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc85f501700 1 --2- 172.21.15.91:0/1537283555 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc8580ce280 0x7fc85814a280 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc85000cba0 tx=0x7fc85000cf60 comp rx=0 tx=0).stop 2023-11-24T06:08:16.051 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc85f501700 1 -- 172.21.15.91:0/1537283555 shutdown_connections 2023-11-24T06:08:16.051 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc85f501700 1 --2- 172.21.15.91:0/1537283555 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc83c06aed0 0x7fc83c06d390 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.051 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc85f501700 1 --2- 172.21.15.91:0/1537283555 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc85815a440 0x7fc8580d0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.051 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc85f501700 1 --2- 172.21.15.91:0/1537283555 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc8580ce280 0x7fc85814a280 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.051 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc85f501700 1 --2- 172.21.15.91:0/1537283555 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc85815eb80 0x7fc8580cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.051 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc85f501700 1 -- 172.21.15.91:0/1537283555 >> 172.21.15.91:0/1537283555 conn(0x7fc8580b5070 msgr2=0x7fc858159750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:16.052 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc85f501700 1 -- 172.21.15.91:0/1537283555 shutdown_connections 2023-11-24T06:08:16.052 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.039+0000 7fc85f501700 1 -- 172.21.15.91:0/1537283555 wait complete. 2023-11-24T06:08:16.055 INFO:teuthology.run_tasks:Running task kclient... 2023-11-24T06:08:16.079 INFO:tasks.kclient:Mounting kernel clients... 2023-11-24T06:08:16.079 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-11-24T06:08:16.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:16.080 DEBUG:teuthology.orchestra.run.smithi091:> ip netns list 2023-11-24T06:08:16.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:16.097 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip link delete ceph-brx 2023-11-24T06:08:16.165 INFO:teuthology.orchestra.run.smithi091.stderr:Cannot find device "ceph-brx" 2023-11-24T06:08:16.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:08:16.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:16.166 DEBUG:teuthology.orchestra.run.smithi091:> ip netns list 2023-11-24T06:08:16.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:16.224 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip link delete ceph-brx 2023-11-24T06:08:16.292 INFO:teuthology.orchestra.run.smithi091.stderr:Cannot find device "ceph-brx" 2023-11-24T06:08:16.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:08:16.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:16.293 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2023-11-24T06:08:16.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:16.315 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link delete ceph-brx 2023-11-24T06:08:16.389 INFO:teuthology.orchestra.run.smithi094.stderr:Cannot find device "ceph-brx" 2023-11-24T06:08:16.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:08:16.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:16.391 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2023-11-24T06:08:16.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:16.449 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link delete ceph-brx 2023-11-24T06:08:16.517 INFO:teuthology.orchestra.run.smithi094.stderr:Cannot find device "ceph-brx" 2023-11-24T06:08:16.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:08:16.519 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2023-11-24T06:08:16.519 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-24T06:08:16.519 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T06:08:16.519 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T06:08:16.640 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.637+0000 7f029fd6b700 1 Processor -- start 2023-11-24T06:08:16.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.637+0000 7f029fd6b700 1 -- start start 2023-11-24T06:08:16.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.638+0000 7f029fd6b700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f029815aa10 0x7f0298072710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.638+0000 7f029fd6b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f02980b0db0 0x7f02980b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.643 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.638+0000 7f029fd6b700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02980af770 con 0x7f029815a630 2023-11-24T06:08:16.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.638+0000 7f029fd6b700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02980ad540 con 0x7f029815aa10 2023-11-24T06:08:16.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.638+0000 7f029fd6b700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029806bfe0 con 0x7f02980b0db0 2023-11-24T06:08:16.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.638+0000 7f029d306700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f029815aa10 0x7f0298072710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.638+0000 7f029d306700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f029815aa10 0x7f0298072710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58474/0 (socket says 172.21.15.91:58474) 2023-11-24T06:08:16.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.638+0000 7f029d306700 1 -- 172.21.15.91:0/2733515745 learned_addr learned my addr 172.21.15.91:0/2733515745 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:16.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.638+0000 7f029e308700 1 --2- 172.21.15.91:0/2733515745 >> v2:172.21.15.94:3300/0 conn(0x7f02980b0db0 0x7f02980b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.638+0000 7f029d306700 1 -- 172.21.15.91:0/2733515745 >> v2:172.21.15.94:3300/0 conn(0x7f02980b0db0 msgr2=0x7f02980b11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:16.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.638+0000 7f029d306700 1 --2- 172.21.15.91:0/2733515745 >> v2:172.21.15.94:3300/0 conn(0x7f02980b0db0 0x7f02980b11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.638+0000 7f029d306700 1 -- 172.21.15.91:0/2733515745 >> v1:172.21.15.94:6789/0 conn(0x7f029815a630 legacy=0x7f02980b06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:16.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.638+0000 7f029d306700 1 -- 172.21.15.91:0/2733515745 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02980afa50 con 0x7f029815aa10 2023-11-24T06:08:16.644 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.638+0000 7f029d306700 1 --2- 172.21.15.91:0/2733515745 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f029815aa10 0x7f0298072710 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0298067900 tx=0x7f028800ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee34981dc50ed780 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:16.645 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.639+0000 7f029cb05700 1 -- 172.21.15.91:0/2733515745 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028800cb30 con 0x7f029815aa10 2023-11-24T06:08:16.645 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.639+0000 7f029cb05700 1 -- 172.21.15.91:0/2733515745 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028800cc90 con 0x7f029815aa10 2023-11-24T06:08:16.645 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.639+0000 7f029cb05700 1 -- 172.21.15.91:0/2733515745 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0288018400 con 0x7f029815aa10 2023-11-24T06:08:16.645 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.639+0000 7f029fd6b700 1 -- 172.21.15.91:0/2733515745 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f029815aa10 msgr2=0x7f0298072710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:16.645 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.639+0000 7f029fd6b700 1 --2- 172.21.15.91:0/2733515745 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f029815aa10 0x7f0298072710 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0298067900 tx=0x7f028800ada0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.645 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.639+0000 7f029fd6b700 1 -- 172.21.15.91:0/2733515745 shutdown_connections 2023-11-24T06:08:16.645 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.639+0000 7f029fd6b700 1 --2- 172.21.15.91:0/2733515745 >> v2:172.21.15.94:3300/0 conn(0x7f02980b0db0 0x7f02980b11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.645 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.639+0000 7f029fd6b700 1 --2- 172.21.15.91:0/2733515745 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f029815aa10 0x7f0298072710 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.645 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.639+0000 7f029fd6b700 1 -- 172.21.15.91:0/2733515745 >> 172.21.15.91:0/2733515745 conn(0x7f02980b3190 msgr2=0x7f02980b35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:16.645 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.639+0000 7f029fd6b700 1 -- 172.21.15.91:0/2733515745 shutdown_connections 2023-11-24T06:08:16.645 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.639+0000 7f029fd6b700 1 -- 172.21.15.91:0/2733515745 wait complete. 2023-11-24T06:08:16.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.640+0000 7f029fd6b700 1 Processor -- start 2023-11-24T06:08:16.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.641+0000 7f029fd6b700 1 -- start start 2023-11-24T06:08:16.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.641+0000 7f029fd6b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02980b0db0 0x7f02981581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.648 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.641+0000 7f029fd6b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f029815a630 0x7f029815c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.641+0000 7f029fd6b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f029815aa10 0x7f02980c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.641+0000 7f029fd6b700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0298158700 con 0x7f029815a630 2023-11-24T06:08:16.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.641+0000 7f029fd6b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029815cb60 con 0x7f02980b0db0 2023-11-24T06:08:16.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.641+0000 7f029fd6b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029815ce40 con 0x7f029815aa10 2023-11-24T06:08:16.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.641+0000 7f029d306700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f029815a630 0x7f029815c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.641+0000 7f029d306700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f029815a630 0x7f029815c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58476/0 (socket says 172.21.15.91:58476) 2023-11-24T06:08:16.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.641+0000 7f029d306700 1 -- 172.21.15.91:0/618509456 learned_addr learned my addr 172.21.15.91:0/618509456 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:16.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.642+0000 7f029db07700 1 --2- 172.21.15.91:0/618509456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02980b0db0 0x7f02981581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.642+0000 7f029d306700 1 -- 172.21.15.91:0/618509456 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f029815aa10 msgr2=0x7f02980c6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:16.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.642+0000 7f029e308700 1 --2- 172.21.15.91:0/618509456 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f029815aa10 0x7f02980c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.642+0000 7f029d306700 1 --2- 172.21.15.91:0/618509456 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f029815aa10 0x7f02980c6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.642+0000 7f029d306700 1 -- 172.21.15.91:0/618509456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02980b0db0 msgr2=0x7f02981581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.642+0000 7f029d306700 1 --2- 172.21.15.91:0/618509456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02980b0db0 0x7f02981581c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.642+0000 7f029d306700 1 -- 172.21.15.91:0/618509456 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f028800a6f0 con 0x7f029815a630 2023-11-24T06:08:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.642+0000 7f029db07700 1 --2- 172.21.15.91:0/618509456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02980b0db0 0x7f02981581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.642+0000 7f029e308700 1 --2- 172.21.15.91:0/618509456 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f029815aa10 0x7f02980c6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.642+0000 7f029d306700 1 --2- 172.21.15.91:0/618509456 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f029815a630 0x7f029815c0c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0288000c00 tx=0x7f02880047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.642+0000 7f028e7fc700 1 -- 172.21.15.91:0/618509456 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0288003b70 con 0x7f029815a630 2023-11-24T06:08:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.642+0000 7f028e7fc700 1 -- 172.21.15.91:0/618509456 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0288003cf0 con 0x7f029815a630 2023-11-24T06:08:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.642+0000 7f029fd6b700 1 -- 172.21.15.91:0/618509456 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029815d120 con 0x7f029815a630 2023-11-24T06:08:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.642+0000 7f028e7fc700 1 -- 172.21.15.91:0/618509456 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0288020640 con 0x7f029815a630 2023-11-24T06:08:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.642+0000 7f029fd6b700 1 -- 172.21.15.91:0/618509456 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02980cd880 con 0x7f029815a630 2023-11-24T06:08:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.643+0000 7f029fd6b700 1 -- 172.21.15.91:0/618509456 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029815d770 con 0x7f029815a630 2023-11-24T06:08:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.643+0000 7f028e7fc700 1 -- 172.21.15.91:0/618509456 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0288018400 con 0x7f029815a630 2023-11-24T06:08:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.644+0000 7f028e7fc700 1 --2- 172.21.15.91:0/618509456 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f027c06aed0 0x7f027c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.644+0000 7f029db07700 1 --2- 172.21.15.91:0/618509456 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f027c06aed0 0x7f027c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.644+0000 7f028e7fc700 1 -- 172.21.15.91:0/618509456 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f028808a9c0 con 0x7f029815a630 2023-11-24T06:08:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.644+0000 7f029db07700 1 --2- 172.21.15.91:0/618509456 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f027c06aed0 0x7f027c06d390 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f02981f7710 tx=0x7f0294009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:16.652 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.648+0000 7f028e7fc700 1 -- 172.21.15.91:0/618509456 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0288055b00 con 0x7f029815a630 2023-11-24T06:08:16.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.883+0000 7f029fd6b700 1 -- 172.21.15.91:0/618509456 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f029815d920 con 0x7f029815a630 2023-11-24T06:08:16.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.884+0000 7f028e7fc700 1 -- 172.21.15.91:0/618509456 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f029815d920 con 0x7f029815a630 2023-11-24T06:08:16.888 INFO:teuthology.orchestra.run.smithi091.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T06:08:16.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.886+0000 7f029fd6b700 1 -- 172.21.15.91:0/618509456 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f027c06aed0 msgr2=0x7f027c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:16.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.886+0000 7f029fd6b700 1 --2- 172.21.15.91:0/618509456 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f027c06aed0 0x7f027c06d390 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f02981f7710 tx=0x7f0294009040 comp rx=0 tx=0).stop 2023-11-24T06:08:16.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.886+0000 7f029fd6b700 1 -- 172.21.15.91:0/618509456 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f029815a630 msgr2=0x7f029815c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:16.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.886+0000 7f029fd6b700 1 --2- 172.21.15.91:0/618509456 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f029815a630 0x7f029815c0c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0288000c00 tx=0x7f02880047c0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.886+0000 7f029fd6b700 1 -- 172.21.15.91:0/618509456 shutdown_connections 2023-11-24T06:08:16.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.886+0000 7f029fd6b700 1 --2- 172.21.15.91:0/618509456 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f027c06aed0 0x7f027c06d390 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.886+0000 7f029fd6b700 1 --2- 172.21.15.91:0/618509456 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02980b0db0 0x7f02981581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.886+0000 7f029fd6b700 1 --2- 172.21.15.91:0/618509456 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f029815aa10 0x7f02980c6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.886+0000 7f029fd6b700 1 --2- 172.21.15.91:0/618509456 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f029815a630 0x7f029815c0c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:16.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.886+0000 7f029fd6b700 1 -- 172.21.15.91:0/618509456 >> 172.21.15.91:0/618509456 conn(0x7f02980b3190 msgr2=0x7f02980c9320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:16.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.886+0000 7f029fd6b700 1 -- 172.21.15.91:0/618509456 shutdown_connections 2023-11-24T06:08:16.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:16.886+0000 7f029fd6b700 1 -- 172.21.15.91:0/618509456 wait complete. 2023-11-24T06:08:16.902 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-24T06:08:16.902 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T06:08:16.902 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi091.front.sepia.ceph.com 2023-11-24T06:08:16.902 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T06:08:16.902 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T06:08:16.902 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-24T06:08:16.903 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-24T06:08:16.903 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T06:08:16.903 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T06:08:16.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:16.903 DEBUG:teuthology.orchestra.run.smithi091:> ip addr 2023-11-24T06:08:16.920 INFO:teuthology.orchestra.run.smithi091.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:08:16.920 INFO:teuthology.orchestra.run.smithi091.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:08:16.920 INFO:teuthology.orchestra.run.smithi091.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:08:16.920 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:16.920 INFO:teuthology.orchestra.run.smithi091.stdout: inet6 ::1/128 scope host 2023-11-24T06:08:16.921 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:16.921 INFO:teuthology.orchestra.run.smithi091.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:08:16.921 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 0c:c4:7a:d8:d9:c4 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:16.921 INFO:teuthology.orchestra.run.smithi091.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:08:16.921 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 0c:c4:7a:d8:d9:c5 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:16.921 INFO:teuthology.orchestra.run.smithi091.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:08:16.921 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 0c:c4:7a:8f:d3:d4 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:16.921 INFO:teuthology.orchestra.run.smithi091.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T06:08:16.921 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 0c:c4:7a:8f:d3:d5 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:16.921 INFO:teuthology.orchestra.run.smithi091.stdout: inet 172.21.15.91/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T06:08:16.922 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft 42083sec preferred_lft 42083sec 2023-11-24T06:08:16.922 INFO:teuthology.orchestra.run.smithi091.stdout: inet6 fe80::f6f7:b098:fe54:3959/64 scope link noprefixroute 2023-11-24T06:08:16.922 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:16.922 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-24T06:08:16.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:16.923 DEBUG:teuthology.orchestra.run.smithi091:> (cd / && exec bash -c ' 2023-11-24T06:08:16.923 DEBUG:teuthology.orchestra.run.smithi091:> set -e 2023-11-24T06:08:16.923 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip link add name ceph-brx type bridge 2023-11-24T06:08:16.923 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip addr flush dev ceph-brx 2023-11-24T06:08:16.923 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip link set ceph-brx up 2023-11-24T06:08:16.923 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-24T06:08:16.923 DEBUG:teuthology.orchestra.run.smithi091:> ') 2023-11-24T06:08:17.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:17.093 DEBUG:teuthology.orchestra.run.smithi091:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T06:08:17.120 INFO:teuthology.orchestra.run.smithi091.stdout:1 2023-11-24T06:08:17.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:17.122 DEBUG:teuthology.orchestra.run.smithi091:> route 2023-11-24T06:08:17.218 INFO:teuthology.orchestra.run.smithi091.stdout:Kernel IP routing table 2023-11-24T06:08:17.218 INFO:teuthology.orchestra.run.smithi091.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T06:08:17.219 INFO:teuthology.orchestra.run.smithi091.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T06:08:17.219 INFO:teuthology.orchestra.run.smithi091.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T06:08:17.219 INFO:teuthology.orchestra.run.smithi091.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T06:08:17.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:17.219 DEBUG:teuthology.orchestra.run.smithi091:> (cd / && exec bash -c ' 2023-11-24T06:08:17.219 DEBUG:teuthology.orchestra.run.smithi091:> set -e 2023-11-24T06:08:17.219 DEBUG:teuthology.orchestra.run.smithi091:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T06:08:17.219 DEBUG:teuthology.orchestra.run.smithi091:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T06:08:17.220 DEBUG:teuthology.orchestra.run.smithi091:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T06:08:17.220 DEBUG:teuthology.orchestra.run.smithi091:> ') 2023-11-24T06:08:17.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:17.449 DEBUG:teuthology.orchestra.run.smithi091:> ip netns list 2023-11-24T06:08:17.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:17.466 DEBUG:teuthology.orchestra.run.smithi091:> ip netns list-id 2023-11-24T06:08:17.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:17.522 DEBUG:teuthology.orchestra.run.smithi091:> (cd / && exec bash -c ' 2023-11-24T06:08:17.522 DEBUG:teuthology.orchestra.run.smithi091:> set -e 2023-11-24T06:08:17.522 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T06:08:17.522 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-11-24T06:08:17.523 DEBUG:teuthology.orchestra.run.smithi091:> ') 2023-11-24T06:08:17.617 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-11-24T06:08:17.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:17.617 DEBUG:teuthology.orchestra.run.smithi091:> (cd / && exec bash -c ' 2023-11-24T06:08:17.617 DEBUG:teuthology.orchestra.run.smithi091:> set -e 2023-11-24T06:08:17.617 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-11-24T06:08:17.617 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-24T06:08:17.617 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-11-24T06:08:17.618 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-11-24T06:08:17.618 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-11-24T06:08:17.618 DEBUG:teuthology.orchestra.run.smithi091:> ') 2023-11-24T06:08:17.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:17.825 DEBUG:teuthology.orchestra.run.smithi091:> (cd / && exec bash -c ' 2023-11-24T06:08:17.825 DEBUG:teuthology.orchestra.run.smithi091:> set -e 2023-11-24T06:08:17.825 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip link set brx.0 up 2023-11-24T06:08:17.825 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip link set dev brx.0 master ceph-brx 2023-11-24T06:08:17.825 DEBUG:teuthology.orchestra.run.smithi091:> ') 2023-11-24T06:08:17.926 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:08:17.927 DEBUG:teuthology.orchestra.run.smithi091:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T06:08:17.946 INFO:teuthology.orchestra.run.smithi091.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T06:08:17.946 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:08:17.946 DEBUG:teuthology.orchestra.run.smithi091:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T06:08:18.001 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T06:08:18.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:18.002 DEBUG:teuthology.orchestra.run.smithi091:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:08:18.100 INFO:teuthology.orchestra.run.smithi091.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:08:18.100 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:08:18.216 INFO:teuthology.orchestra.run.smithi091.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:08:18.216 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:08:18.216 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: resolved to: "172.21.15.91:3300,172.21.15.94:3300,172.21.15.94:3301" 2023-11-24T06:08:18.216 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: trying mount with new device syntax: 0@f7c07073-6cd6-496f-812a-c57c86a2fbbb.cephfs=/ 2023-11-24T06:08:18.216 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.91:3300/172.21.15.94:3300/172.21.15.94:3301" will pass to kernel 2023-11-24T06:08:18.216 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: v2 mount support check returned 2 2023-11-24T06:08:18.216 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: kernel does not support v2 syntax 2023-11-24T06:08:18.216 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:08:18.248 INFO:teuthology.orchestra.run.smithi091.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:08:18.248 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:08:18.248 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: resolved to: "172.21.15.91:3300,172.21.15.94:3300,172.21.15.94:3301" 2023-11-24T06:08:18.248 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: trying mount with new device syntax: 0@f7c07073-6cd6-496f-812a-c57c86a2fbbb.cephfs=/ 2023-11-24T06:08:18.248 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.91:3300/172.21.15.94:3300/172.21.15.94:3301" will pass to kernel 2023-11-24T06:08:18.248 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: v2 mount support check returned 2 2023-11-24T06:08:18.248 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: kernel does not support v2 syntax 2023-11-24T06:08:18.249 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:08:18.249 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: resolved to: "172.21.15.91:6789,172.21.15.94:6789,172.21.15.94:6790" 2023-11-24T06:08:18.249 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.91:6789,172.21.15.94:6789,172.21.15.94:6790:/ 2023-11-24T06:08:18.249 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=f7c07073-6cd6-496f-812a-c57c86a2fbbb" will pass to kernel 2023-11-24T06:08:18.252 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T06:08:18.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:18.252 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T06:08:18.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:18.282 DEBUG:teuthology.orchestra.run.smithi091:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:08:18.363 INFO:teuthology.orchestra.run.smithi091.stdout:client4463 2023-11-24T06:08:18.363 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:08:18.363 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:08:18.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:18.420 DEBUG:teuthology.orchestra.run.smithi091:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:08:18.501 INFO:teuthology.orchestra.run.smithi091.stdout:client4463 2023-11-24T06:08:18.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:18.502 DEBUG:teuthology.orchestra.run.smithi091:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:08:18.582 INFO:teuthology.orchestra.run.smithi091.stdout:client4463 2023-11-24T06:08:18.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:18.583 DEBUG:teuthology.orchestra.run.smithi091:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/f7c07073-6cd6-496f-812a-c57c86a2fbbb.client4463/status') 2023-11-24T06:08:18.671 INFO:teuthology.orchestra.run.smithi091.stdout:instance: client.4463 (3)172.21.15.91:0/562252973 2023-11-24T06:08:18.671 INFO:teuthology.orchestra.run.smithi091.stdout:blocklisted: false 2023-11-24T06:08:18.671 INFO:teuthology.orchestra.run.smithi091.stderr:0+1 records in 2023-11-24T06:08:18.671 INFO:teuthology.orchestra.run.smithi091.stderr:0+1 records out 2023-11-24T06:08:18.671 INFO:teuthology.orchestra.run.smithi091.stderr:69 bytes copied, 2.2804e-05 s, 3.0 MB/s 2023-11-24T06:08:18.672 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2023-11-24T06:08:18.673 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-24T06:08:18.673 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-11-24T06:08:18.673 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T06:08:18.795 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.791+0000 7f3cee895700 1 Processor -- start 2023-11-24T06:08:18.795 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.792+0000 7f3cee895700 1 -- start start 2023-11-24T06:08:18.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.792+0000 7f3cee895700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ce80b5e40 0x7f3ce815ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:18.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.792+0000 7f3cee895700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce806bb80 con 0x7f3ce815e1e0 2023-11-24T06:08:18.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.792+0000 7f3cee895700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce806be20 con 0x7f3ce806b400 2023-11-24T06:08:18.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.792+0000 7f3cee895700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce806c100 con 0x7f3ce80b5e40 2023-11-24T06:08:18.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.792+0000 7f3ce7fff700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f3ce815e1e0 0x7f3ce80b12a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53426/0 (socket says 172.21.15.91:53426) 2023-11-24T06:08:18.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.792+0000 7f3ce7fff700 1 -- 172.21.15.91:0/1936145924 learned_addr learned my addr 172.21.15.91:0/1936145924 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:18.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce6ffd700 1 -- 172.21.15.91:0/1936145924 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451006472 0 0) 0x7f3ce806bb80 con 0x7f3ce815e1e0 2023-11-24T06:08:18.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3cece32700 1 --2- 172.21.15.91:0/1936145924 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ce80b5e40 0x7f3ce815ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:18.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce6ffd700 1 -- 172.21.15.91:0/1936145924 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cd00035b0 con 0x7f3ce815e1e0 2023-11-24T06:08:18.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce6ffd700 1 -- 172.21.15.91:0/1936145924 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1623665751 0 0) 0x7f3cd00035b0 con 0x7f3ce815e1e0 2023-11-24T06:08:18.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce6ffd700 1 -- 172.21.15.91:0/1936145924 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ce80b5e40 msgr2=0x7f3ce815ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:18.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce6ffd700 1 --2- 172.21.15.91:0/1936145924 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ce80b5e40 0x7f3ce815ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:18.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce6ffd700 1 -- 172.21.15.91:0/1936145924 >> v1:172.21.15.94:6789/0 conn(0x7f3ce806b400 legacy=0x7f3ce815e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:18.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce6ffd700 1 -- 172.21.15.91:0/1936145924 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce806c3e0 con 0x7f3ce815e1e0 2023-11-24T06:08:18.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce6ffd700 1 -- 172.21.15.91:0/1936145924 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f3cd8001670 con 0x7f3ce815e1e0 2023-11-24T06:08:18.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce6ffd700 1 -- 172.21.15.91:0/1936145924 >> v1:172.21.15.91:6789/0 conn(0x7f3ce815e1e0 legacy=0x7f3ce80b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:18.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce6ffd700 1 --2- 172.21.15.91:0/1936145924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cd0003a30 0x7f3cd0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:18.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce6ffd700 1 --2- 172.21.15.91:0/1936145924 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3cd0004850 0x7f3cd0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:18.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce6ffd700 1 --2- 172.21.15.91:0/1936145924 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ce80b5e40 0x7f3cd0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:18.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce6ffd700 1 -- 172.21.15.91:0/1936145924 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce806c100 con 0x7f3cd0004850 2023-11-24T06:08:18.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce6ffd700 1 -- 172.21.15.91:0/1936145924 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd000bac0 con 0x7f3cd0003a30 2023-11-24T06:08:18.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce6ffd700 1 -- 172.21.15.91:0/1936145924 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd000bc80 con 0x7f3ce80b5e40 2023-11-24T06:08:18.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce77fe700 1 --2- 172.21.15.91:0/1936145924 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3cd0004850 0x7f3cd0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:18.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce7fff700 1 --2- 172.21.15.91:0/1936145924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cd0003a30 0x7f3cd0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:18.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3cece32700 1 --2- 172.21.15.91:0/1936145924 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ce80b5e40 0x7f3cd0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:18.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce77fe700 1 -- 172.21.15.91:0/1936145924 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ce80b5e40 msgr2=0x7f3cd0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:18.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce77fe700 1 --2- 172.21.15.91:0/1936145924 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3ce80b5e40 0x7f3cd0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:18.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce77fe700 1 -- 172.21.15.91:0/1936145924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cd0003a30 msgr2=0x7f3cd0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:18.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce77fe700 1 --2- 172.21.15.91:0/1936145924 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3cd0003a30 0x7f3cd0003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:18.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.793+0000 7f3ce77fe700 1 -- 172.21.15.91:0/1936145924 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd00035b0 con 0x7f3cd0004850 2023-11-24T06:08:18.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.794+0000 7f3ce77fe700 1 --2- 172.21.15.91:0/1936145924 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3cd0004850 0x7f3cd0004cd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc007fd0 tx=0x7f3cd0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce737d76c6c39c0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:18.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.794+0000 7f3ce6ffd700 1 -- 172.21.15.91:0/1936145924 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc0052d0 con 0x7f3cd0004850 2023-11-24T06:08:18.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.794+0000 7f3ce6ffd700 1 -- 172.21.15.91:0/1936145924 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cdc005430 con 0x7f3cd0004850 2023-11-24T06:08:18.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.794+0000 7f3ce6ffd700 1 -- 172.21.15.91:0/1936145924 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc0038c0 con 0x7f3cd0004850 2023-11-24T06:08:18.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.794+0000 7f3cee895700 1 -- 172.21.15.91:0/1936145924 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3cd0004850 msgr2=0x7f3cd0004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:18.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.794+0000 7f3cee895700 1 --2- 172.21.15.91:0/1936145924 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3cd0004850 0x7f3cd0004cd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc007fd0 tx=0x7f3cd0005880 comp rx=0 tx=0).stop 2023-11-24T06:08:18.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.794+0000 7f3cece32700 1 -- 172.21.15.91:0/1936145924 reap_dead start 2023-11-24T06:08:18.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.794+0000 7f3cee895700 1 -- 172.21.15.91:0/1936145924 shutdown_connections 2023-11-24T06:08:18.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.794+0000 7f3cee895700 1 -- 172.21.15.91:0/1936145924 >> 172.21.15.91:0/1936145924 conn(0x7f3ce80b5070 msgr2=0x7f3ce80b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:18.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.794+0000 7f3cee895700 1 -- 172.21.15.91:0/1936145924 shutdown_connections 2023-11-24T06:08:18.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.794+0000 7f3cee895700 1 -- 172.21.15.91:0/1936145924 wait complete. 2023-11-24T06:08:18.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.795+0000 7f3cee895700 1 Processor -- start 2023-11-24T06:08:18.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.796+0000 7f3cee895700 1 -- start start 2023-11-24T06:08:18.806 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.796+0000 7f3cee895700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cd0004850 0x7f3ce809de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:18.806 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.796+0000 7f3cee895700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ce8159210 0x7f3ce8157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:18.806 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.796+0000 7f3cee895700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3ce806e540 0x7f3ce806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:18.806 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.796+0000 7f3cee895700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce815a390 con 0x7f3ce806e540 2023-11-24T06:08:18.806 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.796+0000 7f3cee895700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce806ef10 con 0x7f3ce8159210 2023-11-24T06:08:18.806 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.796+0000 7f3cee895700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce8159710 con 0x7f3cd0004850 2023-11-24T06:08:18.806 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.796+0000 7f3cece32700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3ce806e540 0x7f3ce806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:18.807 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.797+0000 7f3cece32700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3ce806e540 0x7f3ce806e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58504/0 (socket says 172.21.15.91:58504) 2023-11-24T06:08:18.807 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.797+0000 7f3cece32700 1 -- 172.21.15.91:0/926388073 learned_addr learned my addr 172.21.15.91:0/926388073 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:18.807 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.797+0000 7f3ce7fff700 1 --2- 172.21.15.91:0/926388073 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cd0004850 0x7f3ce809de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:18.807 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.797+0000 7f3ce77fe700 1 --2- 172.21.15.91:0/926388073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ce8159210 0x7f3ce8157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:18.807 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.797+0000 7f3cece32700 1 -- 172.21.15.91:0/926388073 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cd0004850 msgr2=0x7f3ce809de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:18.807 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.797+0000 7f3cece32700 1 --2- 172.21.15.91:0/926388073 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cd0004850 0x7f3ce809de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:18.807 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.797+0000 7f3cece32700 1 -- 172.21.15.91:0/926388073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ce8159210 msgr2=0x7f3ce8157980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:18.807 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.797+0000 7f3cece32700 1 --2- 172.21.15.91:0/926388073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ce8159210 0x7f3ce8157980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:18.807 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.797+0000 7f3cece32700 1 -- 172.21.15.91:0/926388073 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce81599f0 con 0x7f3ce806e540 2023-11-24T06:08:18.808 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.797+0000 7f3ce77fe700 1 --2- 172.21.15.91:0/926388073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ce8159210 0x7f3ce8157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:18.808 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.797+0000 7f3cece32700 1 --2- 172.21.15.91:0/926388073 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3ce806e540 0x7f3ce806e9d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8071590 tx=0x7f3ce000f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:18.808 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.797+0000 7f3ce4ff9700 1 -- 172.21.15.91:0/926388073 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce000e040 con 0x7f3ce806e540 2023-11-24T06:08:18.808 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.797+0000 7f3ce4ff9700 1 -- 172.21.15.91:0/926388073 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce000f040 con 0x7f3ce806e540 2023-11-24T06:08:18.808 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.797+0000 7f3cee895700 1 -- 172.21.15.91:0/926388073 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce8159cd0 con 0x7f3ce806e540 2023-11-24T06:08:18.808 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.797+0000 7f3ce4ff9700 1 -- 172.21.15.91:0/926388073 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce0011e50 con 0x7f3ce806e540 2023-11-24T06:08:18.808 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.797+0000 7f3cee895700 1 -- 172.21.15.91:0/926388073 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce815a920 con 0x7f3ce806e540 2023-11-24T06:08:18.808 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.798+0000 7f3cee895700 1 -- 172.21.15.91:0/926388073 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce80b0610 con 0x7f3ce806e540 2023-11-24T06:08:18.808 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.799+0000 7f3ce4ff9700 1 -- 172.21.15.91:0/926388073 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3ce0015430 con 0x7f3ce806e540 2023-11-24T06:08:18.809 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.799+0000 7f3ce4ff9700 1 --2- 172.21.15.91:0/926388073 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3cd406aed0 0x7f3cd406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:18.809 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.799+0000 7f3ce4ff9700 1 -- 172.21.15.91:0/926388073 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ce0089e40 con 0x7f3ce806e540 2023-11-24T06:08:18.809 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.799+0000 7f3ce7fff700 1 --2- 172.21.15.91:0/926388073 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3cd406aed0 0x7f3cd406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:18.809 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.799+0000 7f3ce7fff700 1 --2- 172.21.15.91:0/926388073 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3cd406aed0 0x7f3cd406d390 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8001c90 tx=0x7f3cd8001b20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:18.809 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:18.804+0000 7f3ce4ff9700 1 -- 172.21.15.91:0/926388073 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ce0054f80 con 0x7f3ce806e540 2023-11-24T06:08:19.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:19.045+0000 7f3cee895700 1 -- 172.21.15.91:0/926388073 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3ce815a4d0 con 0x7f3ce806e540 2023-11-24T06:08:19.049 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:19.046+0000 7f3ce4ff9700 1 -- 172.21.15.91:0/926388073 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3ce815a4d0 con 0x7f3ce806e540 2023-11-24T06:08:19.049 INFO:teuthology.orchestra.run.smithi091.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T06:08:19.051 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:19.048+0000 7f3cee895700 1 -- 172.21.15.91:0/926388073 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3cd406aed0 msgr2=0x7f3cd406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:19.051 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:19.048+0000 7f3cee895700 1 --2- 172.21.15.91:0/926388073 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3cd406aed0 0x7f3cd406d390 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3cd8001c90 tx=0x7f3cd8001b20 comp rx=0 tx=0).stop 2023-11-24T06:08:19.051 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:19.048+0000 7f3cee895700 1 -- 172.21.15.91:0/926388073 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3ce806e540 msgr2=0x7f3ce806e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:19.051 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:19.048+0000 7f3cee895700 1 --2- 172.21.15.91:0/926388073 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3ce806e540 0x7f3ce806e9d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8071590 tx=0x7f3ce000f490 comp rx=0 tx=0).stop 2023-11-24T06:08:19.052 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:19.048+0000 7f3cee895700 1 -- 172.21.15.91:0/926388073 shutdown_connections 2023-11-24T06:08:19.052 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:19.048+0000 7f3cee895700 1 --2- 172.21.15.91:0/926388073 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3cd406aed0 0x7f3cd406d390 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:19.052 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:19.048+0000 7f3cee895700 1 --2- 172.21.15.91:0/926388073 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3ce806e540 0x7f3ce806e9d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:19.052 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:19.048+0000 7f3cee895700 1 --2- 172.21.15.91:0/926388073 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3cd0004850 0x7f3ce809de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:19.052 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:19.048+0000 7f3cee895700 1 --2- 172.21.15.91:0/926388073 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3ce8159210 0x7f3ce8157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:19.052 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:19.048+0000 7f3cee895700 1 -- 172.21.15.91:0/926388073 >> 172.21.15.91:0/926388073 conn(0x7f3ce80b5070 msgr2=0x7f3ce80b1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:19.052 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:19.048+0000 7f3cee895700 1 -- 172.21.15.91:0/926388073 shutdown_connections 2023-11-24T06:08:19.052 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:19.048+0000 7f3cee895700 1 -- 172.21.15.91:0/926388073 wait complete. 2023-11-24T06:08:19.063 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-24T06:08:19.063 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T06:08:19.063 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi091.front.sepia.ceph.com 2023-11-24T06:08:19.063 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-11-24T06:08:19.063 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-11-24T06:08:19.063 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-24T06:08:19.063 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-24T06:08:19.063 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T06:08:19.063 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-11-24T06:08:19.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:19.064 DEBUG:teuthology.orchestra.run.smithi091:> ip addr 2023-11-24T06:08:19.080 INFO:teuthology.orchestra.run.smithi091.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:08:19.081 INFO:teuthology.orchestra.run.smithi091.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:08:19.081 INFO:teuthology.orchestra.run.smithi091.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:08:19.081 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:19.081 INFO:teuthology.orchestra.run.smithi091.stdout: inet6 ::1/128 scope host 2023-11-24T06:08:19.081 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:19.081 INFO:teuthology.orchestra.run.smithi091.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:08:19.081 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 0c:c4:7a:d8:d9:c4 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:19.081 INFO:teuthology.orchestra.run.smithi091.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:08:19.081 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 0c:c4:7a:d8:d9:c5 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:19.081 INFO:teuthology.orchestra.run.smithi091.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:08:19.081 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 0c:c4:7a:8f:d3:d4 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:19.081 INFO:teuthology.orchestra.run.smithi091.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T06:08:19.082 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 0c:c4:7a:8f:d3:d5 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:19.082 INFO:teuthology.orchestra.run.smithi091.stdout: inet 172.21.15.91/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T06:08:19.082 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft 42081sec preferred_lft 42081sec 2023-11-24T06:08:19.082 INFO:teuthology.orchestra.run.smithi091.stdout: inet6 fe80::f6f7:b098:fe54:3959/64 scope link noprefixroute 2023-11-24T06:08:19.082 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:19.082 INFO:teuthology.orchestra.run.smithi091.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:08:19.082 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 76:69:41:d1:ea:d2 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:19.082 INFO:teuthology.orchestra.run.smithi091.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T06:08:19.082 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:19.082 INFO:teuthology.orchestra.run.smithi091.stdout: inet6 fe80::b813:36ff:fe84:b31a/64 scope link 2023-11-24T06:08:19.082 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:19.082 INFO:teuthology.orchestra.run.smithi091.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:08:19.083 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 76:69:41:d1:ea:d2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T06:08:19.083 INFO:teuthology.orchestra.run.smithi091.stdout: inet6 fe80::7469:41ff:fed1:ead2/64 scope link 2023-11-24T06:08:19.083 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:19.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:19.083 DEBUG:teuthology.orchestra.run.smithi091:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T06:08:19.146 INFO:teuthology.orchestra.run.smithi091.stdout:1 2023-11-24T06:08:19.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:19.147 DEBUG:teuthology.orchestra.run.smithi091:> route 2023-11-24T06:08:19.206 INFO:teuthology.orchestra.run.smithi091.stdout:Kernel IP routing table 2023-11-24T06:08:19.206 INFO:teuthology.orchestra.run.smithi091.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T06:08:19.206 INFO:teuthology.orchestra.run.smithi091.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T06:08:19.206 INFO:teuthology.orchestra.run.smithi091.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T06:08:19.206 INFO:teuthology.orchestra.run.smithi091.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T06:08:19.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:19.207 DEBUG:teuthology.orchestra.run.smithi091:> (cd / && exec bash -c ' 2023-11-24T06:08:19.207 DEBUG:teuthology.orchestra.run.smithi091:> set -e 2023-11-24T06:08:19.207 DEBUG:teuthology.orchestra.run.smithi091:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T06:08:19.207 DEBUG:teuthology.orchestra.run.smithi091:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T06:08:19.207 DEBUG:teuthology.orchestra.run.smithi091:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T06:08:19.207 DEBUG:teuthology.orchestra.run.smithi091:> ') 2023-11-24T06:08:19.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:19.311 DEBUG:teuthology.orchestra.run.smithi091:> ip netns list 2023-11-24T06:08:19.326 INFO:teuthology.orchestra.run.smithi091.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T06:08:19.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:19.327 DEBUG:teuthology.orchestra.run.smithi091:> ip netns list-id 2023-11-24T06:08:19.383 INFO:teuthology.orchestra.run.smithi091.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-11-24T06:08:19.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:19.383 DEBUG:teuthology.orchestra.run.smithi091:> (cd / && exec bash -c ' 2023-11-24T06:08:19.384 DEBUG:teuthology.orchestra.run.smithi091:> set -e 2023-11-24T06:08:19.384 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T06:08:19.384 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-11-24T06:08:19.384 DEBUG:teuthology.orchestra.run.smithi091:> ') 2023-11-24T06:08:19.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:19.476 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-11-24T06:08:19.518 INFO:teuthology.orchestra.run.smithi091.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:08:19.518 INFO:teuthology.orchestra.run.smithi091.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:08:19.518 INFO:teuthology.orchestra.run.smithi091.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:08:19.518 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:19.518 INFO:teuthology.orchestra.run.smithi091.stdout: inet6 ::1/128 scope host 2023-11-24T06:08:19.518 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:19.518 INFO:teuthology.orchestra.run.smithi091.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:08:19.518 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 6e:9f:65:59:09:6c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-24T06:08:19.518 INFO:teuthology.orchestra.run.smithi091.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-24T06:08:19.519 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:19.519 INFO:teuthology.orchestra.run.smithi091.stdout: inet6 fe80::6c9f:65ff:fe59:96c/64 scope link 2023-11-24T06:08:19.519 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:19.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:19.529 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-11-24T06:08:19.610 INFO:teuthology.orchestra.run.smithi091.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:08:19.611 INFO:teuthology.orchestra.run.smithi091.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:08:19.611 INFO:teuthology.orchestra.run.smithi091.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:08:19.611 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:19.611 INFO:teuthology.orchestra.run.smithi091.stdout: inet6 ::1/128 scope host 2023-11-24T06:08:19.611 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:19.611 INFO:teuthology.orchestra.run.smithi091.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:08:19.611 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 6e:9f:65:59:09:6c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-24T06:08:19.611 INFO:teuthology.orchestra.run.smithi091.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-24T06:08:19.611 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:19.611 INFO:teuthology.orchestra.run.smithi091.stdout: inet6 fe80::6c9f:65ff:fe59:96c/64 scope link 2023-11-24T06:08:19.611 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:19.623 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-11-24T06:08:19.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:19.623 DEBUG:teuthology.orchestra.run.smithi091:> (cd / && exec bash -c ' 2023-11-24T06:08:19.623 DEBUG:teuthology.orchestra.run.smithi091:> set -e 2023-11-24T06:08:19.624 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-11-24T06:08:19.624 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-11-24T06:08:19.624 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-11-24T06:08:19.624 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-11-24T06:08:19.624 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-11-24T06:08:19.624 DEBUG:teuthology.orchestra.run.smithi091:> ') 2023-11-24T06:08:19.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:19.896 DEBUG:teuthology.orchestra.run.smithi091:> (cd / && exec bash -c ' 2023-11-24T06:08:19.896 DEBUG:teuthology.orchestra.run.smithi091:> set -e 2023-11-24T06:08:19.897 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip link set brx.1 up 2023-11-24T06:08:19.897 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip link set dev brx.1 master ceph-brx 2023-11-24T06:08:19.897 DEBUG:teuthology.orchestra.run.smithi091:> ') 2023-11-24T06:08:20.013 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:08:20.013 DEBUG:teuthology.orchestra.run.smithi091:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-11-24T06:08:20.036 INFO:teuthology.orchestra.run.smithi091.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-11-24T06:08:20.036 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:08:20.037 DEBUG:teuthology.orchestra.run.smithi091:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-11-24T06:08:20.093 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2023-11-24T06:08:20.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:20.093 DEBUG:teuthology.orchestra.run.smithi091:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:08:20.178 INFO:teuthology.orchestra.run.smithi091.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:08:20.178 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:08:20.204 INFO:teuthology.orchestra.run.smithi091.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:08:20.205 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:08:20.205 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: resolved to: "172.21.15.91:3300,172.21.15.94:3300,172.21.15.94:3301" 2023-11-24T06:08:20.205 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: trying mount with new device syntax: 2@f7c07073-6cd6-496f-812a-c57c86a2fbbb.cephfs=/ 2023-11-24T06:08:20.205 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.91:3300/172.21.15.94:3300/172.21.15.94:3301" will pass to kernel 2023-11-24T06:08:20.205 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: v2 mount support check returned 2 2023-11-24T06:08:20.205 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: kernel does not support v2 syntax 2023-11-24T06:08:20.205 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:08:20.235 INFO:teuthology.orchestra.run.smithi091.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:08:20.235 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:08:20.235 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: resolved to: "172.21.15.91:3300,172.21.15.94:3300,172.21.15.94:3301" 2023-11-24T06:08:20.235 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: trying mount with new device syntax: 2@f7c07073-6cd6-496f-812a-c57c86a2fbbb.cephfs=/ 2023-11-24T06:08:20.235 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.91:3300/172.21.15.94:3300/172.21.15.94:3301" will pass to kernel 2023-11-24T06:08:20.235 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: v2 mount support check returned 2 2023-11-24T06:08:20.235 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: kernel does not support v2 syntax 2023-11-24T06:08:20.235 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:08:20.236 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: resolved to: "172.21.15.91:6789,172.21.15.94:6789,172.21.15.94:6790" 2023-11-24T06:08:20.236 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.91:6789,172.21.15.94:6789,172.21.15.94:6790:/ 2023-11-24T06:08:20.236 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,key=2,mds_namespace=cephfs,fsid=f7c07073-6cd6-496f-812a-c57c86a2fbbb" will pass to kernel 2023-11-24T06:08:20.242 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T06:08:20.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:20.243 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-11-24T06:08:20.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:20.275 DEBUG:teuthology.orchestra.run.smithi091:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:08:20.347 INFO:teuthology.orchestra.run.smithi091.stdout:client4674 2023-11-24T06:08:20.347 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:08:20.347 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:08:20.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:20.405 DEBUG:teuthology.orchestra.run.smithi091:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:08:20.476 INFO:teuthology.orchestra.run.smithi091.stdout:client4674 2023-11-24T06:08:20.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:20.477 DEBUG:teuthology.orchestra.run.smithi091:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:08:20.548 INFO:teuthology.orchestra.run.smithi091.stdout:client4674 2023-11-24T06:08:20.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:20.549 DEBUG:teuthology.orchestra.run.smithi091:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/f7c07073-6cd6-496f-812a-c57c86a2fbbb.client4674/status') 2023-11-24T06:08:20.629 INFO:teuthology.orchestra.run.smithi091.stdout:instance: client.4674 (3)192.168.0.2:0/1757992279 2023-11-24T06:08:20.629 INFO:teuthology.orchestra.run.smithi091.stdout:blocklisted: false 2023-11-24T06:08:20.630 INFO:teuthology.orchestra.run.smithi091.stderr:0+1 records in 2023-11-24T06:08:20.630 INFO:teuthology.orchestra.run.smithi091.stderr:0+1 records out 2023-11-24T06:08:20.630 INFO:teuthology.orchestra.run.smithi091.stderr:69 bytes copied, 3.6943e-05 s, 1.9 MB/s 2023-11-24T06:08:20.630 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2023-11-24T06:08:20.631 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-24T06:08:20.631 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T06:08:20.631 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T06:08:20.791 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.788+0000 7f55d509a700 1 Processor -- start 2023-11-24T06:08:20.791 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.788+0000 7f55d509a700 1 -- start start 2023-11-24T06:08:20.793 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.789+0000 7f55d509a700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f55d00cf2b0 0x7f55d015c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:20.794 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.789+0000 7f55d509a700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d00d0a90 con 0x7f55d015bde0 2023-11-24T06:08:20.794 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.789+0000 7f55d509a700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d006ee10 con 0x7f55d00ce5c0 2023-11-24T06:08:20.794 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.789+0000 7f55d509a700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d006f0f0 con 0x7f55d00cf2b0 2023-11-24T06:08:20.794 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.789+0000 7f55ce59c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f55d00cf2b0 0x7f55d015c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:20.794 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.789+0000 7f55ced9d700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f55d015bde0 0x7f55d00d01b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53448/0 (socket says 172.21.15.91:53448) 2023-11-24T06:08:20.794 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.789+0000 7f55ced9d700 1 -- 172.21.15.91:0/2332687877 learned_addr learned my addr 172.21.15.91:0/2332687877 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:20.794 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.789+0000 7f55ce59c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f55d00cf2b0 0x7f55d015c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58516/0 (socket says 172.21.15.91:58516) 2023-11-24T06:08:20.794 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.790+0000 7f55ce59c700 1 -- 172.21.15.91:0/2332687877 >> v1:172.21.15.94:6789/0 conn(0x7f55d00ce5c0 legacy=0x7f55d006fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:20.794 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.790+0000 7f55cdd9b700 1 -- 172.21.15.91:0/2332687877 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701618856 0 0) 0x7f55d00d0a90 con 0x7f55d015bde0 2023-11-24T06:08:20.794 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.790+0000 7f55ce59c700 1 -- 172.21.15.91:0/2332687877 >> v1:172.21.15.91:6789/0 conn(0x7f55d015bde0 legacy=0x7f55d00d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:20.795 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.790+0000 7f55ce59c700 1 -- 172.21.15.91:0/2332687877 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d006f3d0 con 0x7f55d00cf2b0 2023-11-24T06:08:20.795 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.790+0000 7f55ce59c700 1 --2- 172.21.15.91:0/2332687877 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55d00cf2b0 0x7f55d015c1c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009840 tx=0x7f55c0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa0e8ca3ba3ae9a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:20.795 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.790+0000 7f55cdd9b700 1 -- 172.21.15.91:0/2332687877 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c0004230 con 0x7f55d00cf2b0 2023-11-24T06:08:20.795 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.790+0000 7f55cdd9b700 1 -- 172.21.15.91:0/2332687877 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c0004390 con 0x7f55d00cf2b0 2023-11-24T06:08:20.795 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.790+0000 7f55cdd9b700 1 -- 172.21.15.91:0/2332687877 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c00036a0 con 0x7f55d00cf2b0 2023-11-24T06:08:20.795 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.790+0000 7f55d509a700 1 -- 172.21.15.91:0/2332687877 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55d00cf2b0 msgr2=0x7f55d015c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:20.795 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.790+0000 7f55d509a700 1 --2- 172.21.15.91:0/2332687877 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55d00cf2b0 0x7f55d015c1c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009840 tx=0x7f55c0009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:20.795 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.790+0000 7f55d509a700 1 -- 172.21.15.91:0/2332687877 shutdown_connections 2023-11-24T06:08:20.795 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.790+0000 7f55d509a700 1 --2- 172.21.15.91:0/2332687877 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55d00cf2b0 0x7f55d015c1c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:20.795 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.790+0000 7f55d509a700 1 -- 172.21.15.91:0/2332687877 >> 172.21.15.91:0/2332687877 conn(0x7f55d00b3720 msgr2=0x7f55d00b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:20.795 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.790+0000 7f55d509a700 1 -- 172.21.15.91:0/2332687877 shutdown_connections 2023-11-24T06:08:20.795 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.791+0000 7f55d509a700 1 -- 172.21.15.91:0/2332687877 wait complete. 2023-11-24T06:08:20.798 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.792+0000 7f55d509a700 1 Processor -- start 2023-11-24T06:08:20.798 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.792+0000 7f55d509a700 1 -- start start 2023-11-24T06:08:20.798 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55d509a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55d00ce5c0 0x7f55d01f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:20.798 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55d509a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55d00cf2b0 0x7f55d015e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:20.798 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55d509a700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55d015bde0 0x7f55d01583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:20.798 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55d509a700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d01f5ae0 con 0x7f55d015bde0 2023-11-24T06:08:20.798 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55d509a700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d015c380 con 0x7f55d00cf2b0 2023-11-24T06:08:20.798 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55d509a700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d00c7d30 con 0x7f55d00ce5c0 2023-11-24T06:08:20.798 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55cf59e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55d015bde0 0x7f55d01583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:20.798 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55cf59e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55d015bde0 0x7f55d01583f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58524/0 (socket says 172.21.15.91:58524) 2023-11-24T06:08:20.798 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55cf59e700 1 -- 172.21.15.91:0/1270871781 learned_addr learned my addr 172.21.15.91:0/1270871781 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:20.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55ced9d700 1 --2- 172.21.15.91:0/1270871781 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55d00ce5c0 0x7f55d01f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:20.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55ce59c700 1 --2- 172.21.15.91:0/1270871781 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55d00cf2b0 0x7f55d015e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:20.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55cf59e700 1 -- 172.21.15.91:0/1270871781 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55d00ce5c0 msgr2=0x7f55d01f4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:20.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55cf59e700 1 --2- 172.21.15.91:0/1270871781 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55d00ce5c0 0x7f55d01f4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:20.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55cf59e700 1 -- 172.21.15.91:0/1270871781 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55d00cf2b0 msgr2=0x7f55d015e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:20.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55cf59e700 1 --2- 172.21.15.91:0/1270871781 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55d00cf2b0 0x7f55d015e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:20.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55cf59e700 1 -- 172.21.15.91:0/1270871781 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c00094a0 con 0x7f55d015bde0 2023-11-24T06:08:20.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55cf59e700 1 --2- 172.21.15.91:0/1270871781 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55d015bde0 0x7f55d01583f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f55c800cc40 tx=0x7f55c800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:20.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55b77fe700 1 -- 172.21.15.91:0/1270871781 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c800aa80 con 0x7f55d015bde0 2023-11-24T06:08:20.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55b77fe700 1 -- 172.21.15.91:0/1270871781 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c800abe0 con 0x7f55d015bde0 2023-11-24T06:08:20.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55d509a700 1 -- 172.21.15.91:0/1270871781 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d01f5240 con 0x7f55d015bde0 2023-11-24T06:08:20.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.793+0000 7f55b77fe700 1 -- 172.21.15.91:0/1270871781 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c8012420 con 0x7f55d015bde0 2023-11-24T06:08:20.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.794+0000 7f55d509a700 1 -- 172.21.15.91:0/1270871781 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d01f5800 con 0x7f55d015bde0 2023-11-24T06:08:20.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.795+0000 7f55d509a700 1 -- 172.21.15.91:0/1270871781 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d0068670 con 0x7f55d015bde0 2023-11-24T06:08:20.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.795+0000 7f55b77fe700 1 -- 172.21.15.91:0/1270871781 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f55c8012580 con 0x7f55d015bde0 2023-11-24T06:08:20.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.795+0000 7f55b77fe700 1 --2- 172.21.15.91:0/1270871781 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f55bc06aed0 0x7f55bc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:20.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.796+0000 7f55ced9d700 1 --2- 172.21.15.91:0/1270871781 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f55bc06aed0 0x7f55bc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:20.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.796+0000 7f55b77fe700 1 -- 172.21.15.91:0/1270871781 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55c8089c10 con 0x7f55d015bde0 2023-11-24T06:08:20.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.796+0000 7f55ced9d700 1 --2- 172.21.15.91:0/1270871781 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f55bc06aed0 0x7f55bc06d390 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f55c4000f60 tx=0x7f55c4007e70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:20.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:20.800+0000 7f55b77fe700 1 -- 172.21.15.91:0/1270871781 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f55c8054d50 con 0x7f55d015bde0 2023-11-24T06:08:21.034 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:21.032+0000 7f55d509a700 1 -- 172.21.15.91:0/1270871781 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f55d015c4c0 con 0x7f55d015bde0 2023-11-24T06:08:21.035 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:21.032+0000 7f55b77fe700 1 -- 172.21.15.91:0/1270871781 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f55c8016020 con 0x7f55d015bde0 2023-11-24T06:08:21.035 INFO:teuthology.orchestra.run.smithi091.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T06:08:21.037 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:21.034+0000 7f55d509a700 1 -- 172.21.15.91:0/1270871781 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f55bc06aed0 msgr2=0x7f55bc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:21.037 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:21.034+0000 7f55d509a700 1 --2- 172.21.15.91:0/1270871781 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f55bc06aed0 0x7f55bc06d390 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f55c4000f60 tx=0x7f55c4007e70 comp rx=0 tx=0).stop 2023-11-24T06:08:21.038 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:21.034+0000 7f55d509a700 1 -- 172.21.15.91:0/1270871781 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55d015bde0 msgr2=0x7f55d01583f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:21.038 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:21.034+0000 7f55d509a700 1 --2- 172.21.15.91:0/1270871781 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55d015bde0 0x7f55d01583f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f55c800cc40 tx=0x7f55c800cf50 comp rx=0 tx=0).stop 2023-11-24T06:08:21.038 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:21.034+0000 7f55d509a700 1 -- 172.21.15.91:0/1270871781 shutdown_connections 2023-11-24T06:08:21.038 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:21.034+0000 7f55d509a700 1 --2- 172.21.15.91:0/1270871781 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f55bc06aed0 0x7f55bc06d390 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:21.038 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:21.034+0000 7f55d509a700 1 --2- 172.21.15.91:0/1270871781 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55d015bde0 0x7f55d01583f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:21.038 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:21.034+0000 7f55d509a700 1 --2- 172.21.15.91:0/1270871781 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55d00ce5c0 0x7f55d01f4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:21.038 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:21.034+0000 7f55d509a700 1 --2- 172.21.15.91:0/1270871781 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55d00cf2b0 0x7f55d015e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:21.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:21.034+0000 7f55d509a700 1 -- 172.21.15.91:0/1270871781 >> 172.21.15.91:0/1270871781 conn(0x7f55d00b3720 msgr2=0x7f55d00b3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:21.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:21.034+0000 7f55d509a700 1 -- 172.21.15.91:0/1270871781 shutdown_connections 2023-11-24T06:08:21.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:21.034+0000 7f55d509a700 1 -- 172.21.15.91:0/1270871781 wait complete. 2023-11-24T06:08:21.049 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-24T06:08:21.049 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T06:08:21.050 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi094.front.sepia.ceph.com 2023-11-24T06:08:21.050 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T06:08:21.050 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T06:08:21.050 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-24T06:08:21.050 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-24T06:08:21.050 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T06:08:21.050 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T06:08:21.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:21.051 DEBUG:teuthology.orchestra.run.smithi094:> ip addr 2023-11-24T06:08:21.072 INFO:teuthology.orchestra.run.smithi094.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:08:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:08:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:08:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 ::1/128 scope host 2023-11-24T06:08:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:21.073 INFO:teuthology.orchestra.run.smithi094.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:08:21.073 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4c brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:21.073 INFO:teuthology.orchestra.run.smithi094.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:08:21.074 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4d brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:21.074 INFO:teuthology.orchestra.run.smithi094.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:08:21.074 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:62 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:21.074 INFO:teuthology.orchestra.run.smithi094.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T06:08:21.074 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:63 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:21.074 INFO:teuthology.orchestra.run.smithi094.stdout: inet 172.21.15.94/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T06:08:21.074 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft 42074sec preferred_lft 42074sec 2023-11-24T06:08:21.075 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 fe80::d54a:432d:e3af:656/64 scope link noprefixroute 2023-11-24T06:08:21.075 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:21.075 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-11-24T06:08:21.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:21.075 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2023-11-24T06:08:21.075 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2023-11-24T06:08:21.075 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link add name ceph-brx type bridge 2023-11-24T06:08:21.075 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip addr flush dev ceph-brx 2023-11-24T06:08:21.075 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set ceph-brx up 2023-11-24T06:08:21.075 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-11-24T06:08:21.076 DEBUG:teuthology.orchestra.run.smithi094:> ') 2023-11-24T06:08:21.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:21.250 DEBUG:teuthology.orchestra.run.smithi094:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T06:08:21.276 INFO:teuthology.orchestra.run.smithi094.stdout:1 2023-11-24T06:08:21.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:21.278 DEBUG:teuthology.orchestra.run.smithi094:> route 2023-11-24T06:08:21.348 INFO:teuthology.orchestra.run.smithi094.stdout:Kernel IP routing table 2023-11-24T06:08:21.348 INFO:teuthology.orchestra.run.smithi094.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T06:08:21.349 INFO:teuthology.orchestra.run.smithi094.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T06:08:21.349 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T06:08:21.349 INFO:teuthology.orchestra.run.smithi094.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T06:08:21.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:21.349 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2023-11-24T06:08:21.349 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2023-11-24T06:08:21.349 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T06:08:21.349 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T06:08:21.350 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T06:08:21.350 DEBUG:teuthology.orchestra.run.smithi094:> ') 2023-11-24T06:08:21.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:21.628 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2023-11-24T06:08:21.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:21.645 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list-id 2023-11-24T06:08:21.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:21.701 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2023-11-24T06:08:21.701 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2023-11-24T06:08:21.702 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T06:08:21.702 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-11-24T06:08:21.702 DEBUG:teuthology.orchestra.run.smithi094:> ') 2023-11-24T06:08:21.796 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-11-24T06:08:21.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:21.797 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2023-11-24T06:08:21.797 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2023-11-24T06:08:21.797 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-11-24T06:08:21.797 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-11-24T06:08:21.797 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-11-24T06:08:21.797 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-11-24T06:08:21.797 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-11-24T06:08:21.797 DEBUG:teuthology.orchestra.run.smithi094:> ') 2023-11-24T06:08:21.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:21.988 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2023-11-24T06:08:21.988 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2023-11-24T06:08:21.988 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set brx.0 up 2023-11-24T06:08:21.988 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set dev brx.0 master ceph-brx 2023-11-24T06:08:21.988 DEBUG:teuthology.orchestra.run.smithi094:> ') 2023-11-24T06:08:22.086 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:08:22.086 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T06:08:22.106 INFO:teuthology.orchestra.run.smithi094.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T06:08:22.106 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:08:22.106 DEBUG:teuthology.orchestra.run.smithi094:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T06:08:22.162 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T06:08:22.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:22.163 DEBUG:teuthology.orchestra.run.smithi094:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:08:22.254 INFO:teuthology.orchestra.run.smithi094.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:08:22.254 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:08:22.356 INFO:teuthology.orchestra.run.smithi094.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:08:22.357 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:08:22.357 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: resolved to: "172.21.15.91:3300,172.21.15.94:3300,172.21.15.94:3301" 2023-11-24T06:08:22.357 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: trying mount with new device syntax: 1@f7c07073-6cd6-496f-812a-c57c86a2fbbb.cephfs=/ 2023-11-24T06:08:22.357 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.91:3300/172.21.15.94:3300/172.21.15.94:3301" will pass to kernel 2023-11-24T06:08:22.357 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: v2 mount support check returned 2 2023-11-24T06:08:22.357 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: kernel does not support v2 syntax 2023-11-24T06:08:22.357 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:08:22.383 INFO:teuthology.orchestra.run.smithi094.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:08:22.383 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:08:22.383 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: resolved to: "172.21.15.91:3300,172.21.15.94:3300,172.21.15.94:3301" 2023-11-24T06:08:22.383 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: trying mount with new device syntax: 1@f7c07073-6cd6-496f-812a-c57c86a2fbbb.cephfs=/ 2023-11-24T06:08:22.383 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.91:3300/172.21.15.94:3300/172.21.15.94:3301" will pass to kernel 2023-11-24T06:08:22.383 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: v2 mount support check returned 2 2023-11-24T06:08:22.383 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: kernel does not support v2 syntax 2023-11-24T06:08:22.384 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:08:22.384 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: resolved to: "172.21.15.91:6789,172.21.15.94:6789,172.21.15.94:6790" 2023-11-24T06:08:22.384 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.91:6789,172.21.15.94:6789,172.21.15.94:6790:/ 2023-11-24T06:08:22.384 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,key=1,mds_namespace=cephfs,fsid=f7c07073-6cd6-496f-812a-c57c86a2fbbb" will pass to kernel 2023-11-24T06:08:22.387 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T06:08:22.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:22.387 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T06:08:22.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:22.418 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:08:22.490 INFO:teuthology.orchestra.run.smithi094.stdout:client4573 2023-11-24T06:08:22.490 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:08:22.490 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:08:22.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:22.547 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:08:22.619 INFO:teuthology.orchestra.run.smithi094.stdout:client4573 2023-11-24T06:08:22.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:22.619 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:08:22.690 INFO:teuthology.orchestra.run.smithi094.stdout:client4573 2023-11-24T06:08:22.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:22.691 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/f7c07073-6cd6-496f-812a-c57c86a2fbbb.client4573/status') 2023-11-24T06:08:22.770 INFO:teuthology.orchestra.run.smithi094.stdout:instance: client.4573 (3)192.168.0.1:0/2763486164 2023-11-24T06:08:22.771 INFO:teuthology.orchestra.run.smithi094.stdout:blocklisted: false 2023-11-24T06:08:22.771 INFO:teuthology.orchestra.run.smithi094.stderr:0+1 records in 2023-11-24T06:08:22.771 INFO:teuthology.orchestra.run.smithi094.stderr:0+1 records out 2023-11-24T06:08:22.771 INFO:teuthology.orchestra.run.smithi094.stderr:69 bytes copied, 2.1084e-05 s, 3.3 MB/s 2023-11-24T06:08:22.772 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2023-11-24T06:08:22.772 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-11-24T06:08:22.772 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-11-24T06:08:22.772 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T06:08:22.905 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.902+0000 7f9f7121b700 1 Processor -- start 2023-11-24T06:08:22.905 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.902+0000 7f9f7121b700 1 -- start start 2023-11-24T06:08:22.907 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.903+0000 7f9f7121b700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f9f6c06b620 0x7f9f6c0b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:22.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.903+0000 7f9f7121b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9f6c0b5ab0 0x7f9f6c0b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:22.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.903+0000 7f9f7121b700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6c071f50 con 0x7f9f6c1478c0 2023-11-24T06:08:22.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.903+0000 7f9f7121b700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f6c072230 con 0x7f9f6c06b620 2023-11-24T06:08:22.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.903+0000 7f9f7121b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c072510 con 0x7f9f6c0b5ab0 2023-11-24T06:08:22.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.903+0000 7f9f6ad9d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f9f6c06b620 0x7f9f6c0b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:22.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.903+0000 7f9f6ad9d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f9f6c06b620 0x7f9f6c0b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58528/0 (socket says 172.21.15.91:58528) 2023-11-24T06:08:22.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.903+0000 7f9f6ad9d700 1 -- 172.21.15.91:0/181894888 learned_addr learned my addr 172.21.15.91:0/181894888 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:22.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.903+0000 7f9f6a59c700 1 --2- 172.21.15.91:0/181894888 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9f6c0b5ab0 0x7f9f6c0b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:22.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.903+0000 7f9f6ad9d700 1 -- 172.21.15.91:0/181894888 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9f6c0b5ab0 msgr2=0x7f9f6c0b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:22.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.903+0000 7f9f6ad9d700 1 --2- 172.21.15.91:0/181894888 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9f6c0b5ab0 0x7f9f6c0b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:22.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.903+0000 7f9f6ad9d700 1 -- 172.21.15.91:0/181894888 >> v1:172.21.15.94:6789/0 conn(0x7f9f6c1478c0 legacy=0x7f9f6c161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:22.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.903+0000 7f9f69d9b700 1 -- 172.21.15.91:0/181894888 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791281759 0 0) 0x7f9f6c071f50 con 0x7f9f6c1478c0 2023-11-24T06:08:22.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.903+0000 7f9f6ad9d700 1 -- 172.21.15.91:0/181894888 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c0727f0 con 0x7f9f6c06b620 2023-11-24T06:08:22.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.904+0000 7f9f6ad9d700 1 --2- 172.21.15.91:0/181894888 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9f6c06b620 0x7f9f6c0b3a40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9f600098f0 tx=0x7f9f60009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6be9a44b409d959d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:22.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.904+0000 7f9f69d9b700 1 -- 172.21.15.91:0/181894888 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f60011830 con 0x7f9f6c06b620 2023-11-24T06:08:22.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.904+0000 7f9f69d9b700 1 -- 172.21.15.91:0/181894888 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f60011990 con 0x7f9f6c06b620 2023-11-24T06:08:22.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.904+0000 7f9f69d9b700 1 -- 172.21.15.91:0/181894888 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f60011d20 con 0x7f9f6c06b620 2023-11-24T06:08:22.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.904+0000 7f9f7121b700 1 -- 172.21.15.91:0/181894888 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9f6c06b620 msgr2=0x7f9f6c0b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:22.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.904+0000 7f9f7121b700 1 --2- 172.21.15.91:0/181894888 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9f6c06b620 0x7f9f6c0b3a40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9f600098f0 tx=0x7f9f60009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:22.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.904+0000 7f9f7121b700 1 -- 172.21.15.91:0/181894888 shutdown_connections 2023-11-24T06:08:22.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.904+0000 7f9f7121b700 1 --2- 172.21.15.91:0/181894888 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9f6c06b620 0x7f9f6c0b3a40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:22.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.904+0000 7f9f7121b700 1 --2- 172.21.15.91:0/181894888 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9f6c0b5ab0 0x7f9f6c0b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:22.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.904+0000 7f9f7121b700 1 -- 172.21.15.91:0/181894888 >> 172.21.15.91:0/181894888 conn(0x7f9f6c0b2740 msgr2=0x7f9f6c0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:22.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.904+0000 7f9f7121b700 1 -- 172.21.15.91:0/181894888 shutdown_connections 2023-11-24T06:08:22.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.904+0000 7f9f7121b700 1 -- 172.21.15.91:0/181894888 wait complete. 2023-11-24T06:08:22.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.906+0000 7f9f7121b700 1 Processor -- start 2023-11-24T06:08:22.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.906+0000 7f9f7121b700 1 -- start start 2023-11-24T06:08:22.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f7121b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9f6c06b620 0x7f9f6c1e62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:22.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f7121b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9f6c0b5ab0 0x7f9f6c0c5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:22.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f7121b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f6c1478c0 0x7f9f6c0c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:22.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f7121b700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c0c6080 con 0x7f9f6c0b5ab0 2023-11-24T06:08:22.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f7121b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c1e7200 con 0x7f9f6c1478c0 2023-11-24T06:08:22.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f7121b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c1e82e0 con 0x7f9f6c06b620 2023-11-24T06:08:22.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f6a59c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9f6c0b5ab0 0x7f9f6c0c5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:22.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f6a59c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9f6c0b5ab0 0x7f9f6c0c5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58536/0 (socket says 172.21.15.91:58536) 2023-11-24T06:08:22.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f6a59c700 1 -- 172.21.15.91:0/3644847042 learned_addr learned my addr 172.21.15.91:0/3644847042 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:22.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f6ad9d700 1 --2- 172.21.15.91:0/3644847042 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9f6c06b620 0x7f9f6c1e62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:22.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f6b59e700 1 --2- 172.21.15.91:0/3644847042 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f6c1478c0 0x7f9f6c0c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:22.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f6a59c700 1 -- 172.21.15.91:0/3644847042 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9f6c06b620 msgr2=0x7f9f6c1e62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:22.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f6a59c700 1 --2- 172.21.15.91:0/3644847042 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9f6c06b620 0x7f9f6c1e62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:22.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f6a59c700 1 -- 172.21.15.91:0/3644847042 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f6c1478c0 msgr2=0x7f9f6c0c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:22.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f6a59c700 1 --2- 172.21.15.91:0/3644847042 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f6c1478c0 0x7f9f6c0c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:22.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f6a59c700 1 -- 172.21.15.91:0/3644847042 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f60009550 con 0x7f9f6c0b5ab0 2023-11-24T06:08:22.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.907+0000 7f9f6a59c700 1 --2- 172.21.15.91:0/3644847042 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9f6c0b5ab0 0x7f9f6c0c5b40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c00e9b0 tx=0x7f9f5c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:22.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.908+0000 7f9f537fe700 1 -- 172.21.15.91:0/3644847042 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c009710 con 0x7f9f6c0b5ab0 2023-11-24T06:08:22.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.908+0000 7f9f537fe700 1 -- 172.21.15.91:0/3644847042 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5c00ee50 con 0x7f9f6c0b5ab0 2023-11-24T06:08:22.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.908+0000 7f9f7121b700 1 -- 172.21.15.91:0/3644847042 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6c1e6960 con 0x7f9f6c0b5ab0 2023-11-24T06:08:22.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.908+0000 7f9f537fe700 1 -- 172.21.15.91:0/3644847042 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c00ca90 con 0x7f9f6c0b5ab0 2023-11-24T06:08:22.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.908+0000 7f9f7121b700 1 -- 172.21.15.91:0/3644847042 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6c1e6f20 con 0x7f9f6c0b5ab0 2023-11-24T06:08:22.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.908+0000 7f9f7121b700 1 -- 172.21.15.91:0/3644847042 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6c1e8470 con 0x7f9f6c0b5ab0 2023-11-24T06:08:22.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.909+0000 7f9f537fe700 1 -- 172.21.15.91:0/3644847042 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9f5c00cbf0 con 0x7f9f6c0b5ab0 2023-11-24T06:08:22.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.910+0000 7f9f537fe700 1 --2- 172.21.15.91:0/3644847042 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9f5806aed0 0x7f9f5806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:22.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.910+0000 7f9f537fe700 1 -- 172.21.15.91:0/3644847042 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f5c089c20 con 0x7f9f6c0b5ab0 2023-11-24T06:08:22.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.910+0000 7f9f6ad9d700 1 --2- 172.21.15.91:0/3644847042 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9f5806aed0 0x7f9f5806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:22.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.910+0000 7f9f6ad9d700 1 --2- 172.21.15.91:0/3644847042 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9f5806aed0 0x7f9f5806d390 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9f60000c00 tx=0x7f9f60008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:22.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:22.914+0000 7f9f537fe700 1 -- 172.21.15.91:0/3644847042 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f5c054e40 con 0x7f9f6c0b5ab0 2023-11-24T06:08:23.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:23.151+0000 7f9f7121b700 1 -- 172.21.15.91:0/3644847042 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9f6c068670 con 0x7f9f6c0b5ab0 2023-11-24T06:08:23.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:23.151+0000 7f9f537fe700 1 -- 172.21.15.91:0/3644847042 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9f5c017070 con 0x7f9f6c0b5ab0 2023-11-24T06:08:23.154 INFO:teuthology.orchestra.run.smithi091.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T06:08:23.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:23.153+0000 7f9f7121b700 1 -- 172.21.15.91:0/3644847042 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9f5806aed0 msgr2=0x7f9f5806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:23.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:23.153+0000 7f9f7121b700 1 --2- 172.21.15.91:0/3644847042 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9f5806aed0 0x7f9f5806d390 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9f60000c00 tx=0x7f9f60008030 comp rx=0 tx=0).stop 2023-11-24T06:08:23.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:23.153+0000 7f9f7121b700 1 -- 172.21.15.91:0/3644847042 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9f6c0b5ab0 msgr2=0x7f9f6c0c5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:23.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:23.153+0000 7f9f7121b700 1 --2- 172.21.15.91:0/3644847042 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9f6c0b5ab0 0x7f9f6c0c5b40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c00e9b0 tx=0x7f9f5c00ecc0 comp rx=0 tx=0).stop 2023-11-24T06:08:23.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:23.153+0000 7f9f7121b700 1 -- 172.21.15.91:0/3644847042 shutdown_connections 2023-11-24T06:08:23.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:23.153+0000 7f9f7121b700 1 --2- 172.21.15.91:0/3644847042 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f9f5806aed0 0x7f9f5806d390 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:23.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:23.153+0000 7f9f7121b700 1 --2- 172.21.15.91:0/3644847042 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9f6c1478c0 0x7f9f6c0c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:23.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:23.153+0000 7f9f7121b700 1 --2- 172.21.15.91:0/3644847042 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9f6c06b620 0x7f9f6c1e62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:23.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:23.153+0000 7f9f7121b700 1 --2- 172.21.15.91:0/3644847042 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9f6c0b5ab0 0x7f9f6c0c5b40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:23.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:23.153+0000 7f9f7121b700 1 -- 172.21.15.91:0/3644847042 >> 172.21.15.91:0/3644847042 conn(0x7f9f6c0b2740 msgr2=0x7f9f6c0b0770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:23.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:23.153+0000 7f9f7121b700 1 -- 172.21.15.91:0/3644847042 shutdown_connections 2023-11-24T06:08:23.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:23.153+0000 7f9f7121b700 1 -- 172.21.15.91:0/3644847042 wait complete. 2023-11-24T06:08:23.168 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-11-24T06:08:23.168 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T06:08:23.168 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi094.front.sepia.ceph.com 2023-11-24T06:08:23.168 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-11-24T06:08:23.168 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-11-24T06:08:23.168 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-11-24T06:08:23.168 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-11-24T06:08:23.169 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T06:08:23.169 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-11-24T06:08:23.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:23.169 DEBUG:teuthology.orchestra.run.smithi094:> ip addr 2023-11-24T06:08:23.187 INFO:teuthology.orchestra.run.smithi094.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:08:23.187 INFO:teuthology.orchestra.run.smithi094.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:08:23.187 INFO:teuthology.orchestra.run.smithi094.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:08:23.187 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:23.187 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 ::1/128 scope host 2023-11-24T06:08:23.187 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:23.187 INFO:teuthology.orchestra.run.smithi094.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:08:23.187 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4c brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:23.187 INFO:teuthology.orchestra.run.smithi094.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:08:23.188 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4d brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:23.188 INFO:teuthology.orchestra.run.smithi094.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:08:23.188 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:62 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:23.188 INFO:teuthology.orchestra.run.smithi094.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T06:08:23.188 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:63 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:23.188 INFO:teuthology.orchestra.run.smithi094.stdout: inet 172.21.15.94/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T06:08:23.188 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft 42072sec preferred_lft 42072sec 2023-11-24T06:08:23.188 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 fe80::d54a:432d:e3af:656/64 scope link noprefixroute 2023-11-24T06:08:23.188 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:23.188 INFO:teuthology.orchestra.run.smithi094.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:08:23.188 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0a:12:45:d5:24:96 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:08:23.189 INFO:teuthology.orchestra.run.smithi094.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T06:08:23.189 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:23.189 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 fe80::84ef:82ff:fe13:f3ab/64 scope link 2023-11-24T06:08:23.189 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:23.189 INFO:teuthology.orchestra.run.smithi094.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:08:23.189 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0a:12:45:d5:24:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T06:08:23.189 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 fe80::812:45ff:fed5:2496/64 scope link tentative 2023-11-24T06:08:23.189 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:23.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:23.190 DEBUG:teuthology.orchestra.run.smithi094:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T06:08:23.253 INFO:teuthology.orchestra.run.smithi094.stdout:1 2023-11-24T06:08:23.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:23.254 DEBUG:teuthology.orchestra.run.smithi094:> route 2023-11-24T06:08:23.312 INFO:teuthology.orchestra.run.smithi094.stdout:Kernel IP routing table 2023-11-24T06:08:23.313 INFO:teuthology.orchestra.run.smithi094.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T06:08:23.313 INFO:teuthology.orchestra.run.smithi094.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T06:08:23.313 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T06:08:23.313 INFO:teuthology.orchestra.run.smithi094.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T06:08:23.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:23.313 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2023-11-24T06:08:23.313 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2023-11-24T06:08:23.313 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T06:08:23.314 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T06:08:23.314 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T06:08:23.314 DEBUG:teuthology.orchestra.run.smithi094:> ') 2023-11-24T06:08:23.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:23.418 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2023-11-24T06:08:23.434 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T06:08:23.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:23.435 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list-id 2023-11-24T06:08:23.491 INFO:teuthology.orchestra.run.smithi094.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-11-24T06:08:23.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:23.491 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2023-11-24T06:08:23.491 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2023-11-24T06:08:23.492 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T06:08:23.492 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-11-24T06:08:23.492 DEBUG:teuthology.orchestra.run.smithi094:> ') 2023-11-24T06:08:23.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:23.583 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-11-24T06:08:23.617 INFO:teuthology.orchestra.run.smithi094.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:08:23.617 INFO:teuthology.orchestra.run.smithi094.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:08:23.617 INFO:teuthology.orchestra.run.smithi094.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:08:23.617 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:23.617 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 ::1/128 scope host 2023-11-24T06:08:23.617 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:23.617 INFO:teuthology.orchestra.run.smithi094.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:08:23.618 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether e6:9f:42:af:4d:9e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-24T06:08:23.618 INFO:teuthology.orchestra.run.smithi094.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-24T06:08:23.618 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:23.618 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 fe80::e49f:42ff:feaf:4d9e/64 scope link tentative 2023-11-24T06:08:23.618 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:23.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:23.629 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-11-24T06:08:23.709 INFO:teuthology.orchestra.run.smithi094.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:08:23.709 INFO:teuthology.orchestra.run.smithi094.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:08:23.709 INFO:teuthology.orchestra.run.smithi094.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:08:23.709 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:23.709 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 ::1/128 scope host 2023-11-24T06:08:23.709 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:23.709 INFO:teuthology.orchestra.run.smithi094.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:08:23.709 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether e6:9f:42:af:4d:9e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-11-24T06:08:23.710 INFO:teuthology.orchestra.run.smithi094.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-11-24T06:08:23.710 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:23.710 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 fe80::e49f:42ff:feaf:4d9e/64 scope link tentative 2023-11-24T06:08:23.710 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:08:23.719 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-11-24T06:08:23.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:23.719 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2023-11-24T06:08:23.720 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2023-11-24T06:08:23.720 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-11-24T06:08:23.720 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-11-24T06:08:23.720 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-11-24T06:08:23.720 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-11-24T06:08:23.720 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-11-24T06:08:23.720 DEBUG:teuthology.orchestra.run.smithi094:> ') 2023-11-24T06:08:23.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:23.973 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2023-11-24T06:08:23.973 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2023-11-24T06:08:23.973 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set brx.1 up 2023-11-24T06:08:23.974 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set dev brx.1 master ceph-brx 2023-11-24T06:08:23.974 DEBUG:teuthology.orchestra.run.smithi094:> ') 2023-11-24T06:08:24.082 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:08:24.083 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-11-24T06:08:24.102 INFO:teuthology.orchestra.run.smithi094.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-11-24T06:08:24.102 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:08:24.102 DEBUG:teuthology.orchestra.run.smithi094:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-11-24T06:08:24.157 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2023-11-24T06:08:24.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:24.158 DEBUG:teuthology.orchestra.run.smithi094:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:08:24.243 INFO:teuthology.orchestra.run.smithi094.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:08:24.243 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:08:24.266 INFO:teuthology.orchestra.run.smithi094.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:08:24.266 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:08:24.266 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: resolved to: "172.21.15.91:3300,172.21.15.94:3300,172.21.15.94:3301" 2023-11-24T06:08:24.266 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: trying mount with new device syntax: 3@f7c07073-6cd6-496f-812a-c57c86a2fbbb.cephfs=/ 2023-11-24T06:08:24.266 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.91:3300/172.21.15.94:3300/172.21.15.94:3301" will pass to kernel 2023-11-24T06:08:24.266 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: v2 mount support check returned 2 2023-11-24T06:08:24.266 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: kernel does not support v2 syntax 2023-11-24T06:08:24.266 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:08:24.297 INFO:teuthology.orchestra.run.smithi094.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:08:24.297 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:08:24.297 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: resolved to: "172.21.15.91:3300,172.21.15.94:3300,172.21.15.94:3301" 2023-11-24T06:08:24.297 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: trying mount with new device syntax: 3@f7c07073-6cd6-496f-812a-c57c86a2fbbb.cephfs=/ 2023-11-24T06:08:24.297 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.91:3300/172.21.15.94:3300/172.21.15.94:3301" will pass to kernel 2023-11-24T06:08:24.297 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: v2 mount support check returned 2 2023-11-24T06:08:24.297 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: kernel does not support v2 syntax 2023-11-24T06:08:24.297 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:08:24.297 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: resolved to: "172.21.15.91:6789,172.21.15.94:6789,172.21.15.94:6790" 2023-11-24T06:08:24.297 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.91:6789,172.21.15.94:6789,172.21.15.94:6790:/ 2023-11-24T06:08:24.298 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,key=3,mds_namespace=cephfs,fsid=f7c07073-6cd6-496f-812a-c57c86a2fbbb" will pass to kernel 2023-11-24T06:08:24.299 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T06:08:24.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:24.300 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-11-24T06:08:24.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:24.345 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:08:24.416 INFO:teuthology.orchestra.run.smithi094.stdout:client4582 2023-11-24T06:08:24.417 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:08:24.417 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:08:24.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:24.474 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:08:24.547 INFO:teuthology.orchestra.run.smithi094.stdout:client4582 2023-11-24T06:08:24.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:24.547 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:08:24.619 INFO:teuthology.orchestra.run.smithi094.stdout:client4582 2023-11-24T06:08:24.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:24.619 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/f7c07073-6cd6-496f-812a-c57c86a2fbbb.client4582/status') 2023-11-24T06:08:24.699 INFO:teuthology.orchestra.run.smithi094.stdout:instance: client.4582 (3)192.168.0.2:0/2229372764 2023-11-24T06:08:24.699 INFO:teuthology.orchestra.run.smithi094.stdout:blocklisted: false 2023-11-24T06:08:24.700 INFO:teuthology.orchestra.run.smithi094.stderr:0+1 records in 2023-11-24T06:08:24.700 INFO:teuthology.orchestra.run.smithi094.stderr:0+1 records out 2023-11-24T06:08:24.700 INFO:teuthology.orchestra.run.smithi094.stderr:69 bytes copied, 2.0349e-05 s, 3.4 MB/s 2023-11-24T06:08:24.701 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-11-24T06:08:24.719 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:24.853 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.850+0000 7f4abc904700 1 Processor -- start 2023-11-24T06:08:24.854 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.851+0000 7f4abc904700 1 -- start start 2023-11-24T06:08:24.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.852+0000 7f4abc904700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4ab80b5ab0 0x7f4ab806e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:24.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.852+0000 7f4abc904700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab8071f50 con 0x7f4ab80b4150 2023-11-24T06:08:24.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.852+0000 7f4abc904700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab8072230 con 0x7f4ab806b620 2023-11-24T06:08:24.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.852+0000 7f4abc904700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab8072510 con 0x7f4ab80b5ab0 2023-11-24T06:08:24.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.852+0000 7f4ab5d9b700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f4ab80b4150 0x7f4ab806de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53478/0 (socket says 172.21.15.91:53478) 2023-11-24T06:08:24.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.852+0000 7f4ab5d9b700 1 -- 172.21.15.91:0/185758152 learned_addr learned my addr 172.21.15.91:0/185758152 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:24.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.852+0000 7f4ab6d9d700 1 --2- 172.21.15.91:0/185758152 >> v2:172.21.15.94:3300/0 conn(0x7f4ab80b5ab0 0x7f4ab806e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:24.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.852+0000 7f4ab559a700 1 -- 172.21.15.91:0/185758152 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159314154 0 0) 0x7f4ab8071f50 con 0x7f4ab80b4150 2023-11-24T06:08:24.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.852+0000 7f4ab559a700 1 -- 172.21.15.91:0/185758152 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa00035b0 con 0x7f4ab80b4150 2023-11-24T06:08:24.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.852+0000 7f4ab559a700 1 -- 172.21.15.91:0/185758152 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011382486 0 0) 0x7f4ab8072230 con 0x7f4ab806b620 2023-11-24T06:08:24.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.852+0000 7f4ab559a700 1 -- 172.21.15.91:0/185758152 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ab8071f50 con 0x7f4ab806b620 2023-11-24T06:08:24.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.852+0000 7f4ab6d9d700 1 -- 172.21.15.91:0/185758152 >> v1:172.21.15.94:6789/0 conn(0x7f4ab806b620 legacy=0x7f4ab80b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:24.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.852+0000 7f4ab6d9d700 1 -- 172.21.15.91:0/185758152 >> v1:172.21.15.91:6789/0 conn(0x7f4ab80b4150 legacy=0x7f4ab806de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:24.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.852+0000 7f4ab6d9d700 1 -- 172.21.15.91:0/185758152 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab80727f0 con 0x7f4ab80b5ab0 2023-11-24T06:08:24.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.852+0000 7f4ab6d9d700 1 --2- 172.21.15.91:0/185758152 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab80b5ab0 0x7f4ab806e560 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0009d90 tx=0x7f4ab000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4c9c25780ca095 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:24.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.853+0000 7f4ab559a700 1 -- 172.21.15.91:0/185758152 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab000e070 con 0x7f4ab80b5ab0 2023-11-24T06:08:24.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.853+0000 7f4ab559a700 1 -- 172.21.15.91:0/185758152 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab000bb20 con 0x7f4ab80b5ab0 2023-11-24T06:08:24.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.853+0000 7f4ab559a700 1 -- 172.21.15.91:0/185758152 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0012430 con 0x7f4ab80b5ab0 2023-11-24T06:08:24.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.853+0000 7f4abc904700 1 -- 172.21.15.91:0/185758152 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab80b5ab0 msgr2=0x7f4ab806e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:24.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.853+0000 7f4abc904700 1 --2- 172.21.15.91:0/185758152 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab80b5ab0 0x7f4ab806e560 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0009d90 tx=0x7f4ab000b330 comp rx=0 tx=0).stop 2023-11-24T06:08:24.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.853+0000 7f4abc904700 1 -- 172.21.15.91:0/185758152 shutdown_connections 2023-11-24T06:08:24.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.853+0000 7f4abc904700 1 --2- 172.21.15.91:0/185758152 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab80b5ab0 0x7f4ab806e560 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:24.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.853+0000 7f4abc904700 1 -- 172.21.15.91:0/185758152 >> 172.21.15.91:0/185758152 conn(0x7f4ab80b2740 msgr2=0x7f4ab80b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:24.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.853+0000 7f4abc904700 1 -- 172.21.15.91:0/185758152 shutdown_connections 2023-11-24T06:08:24.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.853+0000 7f4abc904700 1 -- 172.21.15.91:0/185758152 wait complete. 2023-11-24T06:08:24.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.855+0000 7f4abc904700 1 Processor -- start 2023-11-24T06:08:24.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.855+0000 7f4abc904700 1 -- start start 2023-11-24T06:08:24.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.855+0000 7f4abc904700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab806b620 0x7f4ab81e5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:24.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.855+0000 7f4abc904700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ab80b4150 0x7f4ab80ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:24.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.855+0000 7f4abc904700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ab80b5ab0 0x7f4ab80cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:24.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.855+0000 7f4abc904700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab8133210 con 0x7f4ab80b4150 2023-11-24T06:08:24.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.855+0000 7f4abc904700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab81e5780 con 0x7f4ab806b620 2023-11-24T06:08:24.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.855+0000 7f4abc904700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab8133460 con 0x7f4ab80b5ab0 2023-11-24T06:08:24.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.855+0000 7f4ab5d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ab80b4150 0x7f4ab80ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:24.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.856+0000 7f4ab5d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ab80b4150 0x7f4ab80ca630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58550/0 (socket says 172.21.15.91:58550) 2023-11-24T06:08:24.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.856+0000 7f4ab5d9b700 1 -- 172.21.15.91:0/3300056677 learned_addr learned my addr 172.21.15.91:0/3300056677 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:24.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.856+0000 7f4ab659c700 1 --2- 172.21.15.91:0/3300056677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab806b620 0x7f4ab81e5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:24.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.856+0000 7f4ab6d9d700 1 --2- 172.21.15.91:0/3300056677 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ab80b5ab0 0x7f4ab80cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:24.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.856+0000 7f4ab5d9b700 1 -- 172.21.15.91:0/3300056677 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ab80b5ab0 msgr2=0x7f4ab80cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:24.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.856+0000 7f4ab5d9b700 1 --2- 172.21.15.91:0/3300056677 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ab80b5ab0 0x7f4ab80cab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:24.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.856+0000 7f4ab5d9b700 1 -- 172.21.15.91:0/3300056677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab806b620 msgr2=0x7f4ab81e5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:24.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.856+0000 7f4ab5d9b700 1 --2- 172.21.15.91:0/3300056677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab806b620 0x7f4ab81e5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:24.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.856+0000 7f4ab5d9b700 1 -- 172.21.15.91:0/3300056677 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab0009a70 con 0x7f4ab80b4150 2023-11-24T06:08:24.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.856+0000 7f4ab5d9b700 1 --2- 172.21.15.91:0/3300056677 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ab80b4150 0x7f4ab80ca630 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4aac00c700 tx=0x7f4aac00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:24.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.856+0000 7f4aa6ffd700 1 -- 172.21.15.91:0/3300056677 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac005d70 con 0x7f4ab80b4150 2023-11-24T06:08:24.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.856+0000 7f4aa6ffd700 1 -- 172.21.15.91:0/3300056677 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aac0070d0 con 0x7f4ab80b4150 2023-11-24T06:08:24.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.856+0000 7f4abc904700 1 -- 172.21.15.91:0/3300056677 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab8133740 con 0x7f4ab80b4150 2023-11-24T06:08:24.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.856+0000 7f4aa6ffd700 1 -- 172.21.15.91:0/3300056677 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac007460 con 0x7f4ab80b4150 2023-11-24T06:08:24.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.856+0000 7f4abc904700 1 -- 172.21.15.91:0/3300056677 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab8133d00 con 0x7f4ab80b4150 2023-11-24T06:08:24.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.857+0000 7f4abc904700 1 -- 172.21.15.91:0/3300056677 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab8134070 con 0x7f4ab80b4150 2023-11-24T06:08:24.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.858+0000 7f4aa6ffd700 1 -- 172.21.15.91:0/3300056677 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4aac0075c0 con 0x7f4ab80b4150 2023-11-24T06:08:24.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.858+0000 7f4aa6ffd700 1 --2- 172.21.15.91:0/3300056677 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4a9806aed0 0x7f4a9806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:24.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.858+0000 7f4aa6ffd700 1 -- 172.21.15.91:0/3300056677 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4aac088a20 con 0x7f4ab80b4150 2023-11-24T06:08:24.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.858+0000 7f4ab659c700 1 --2- 172.21.15.91:0/3300056677 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4a9806aed0 0x7f4a9806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:24.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.858+0000 7f4ab659c700 1 --2- 172.21.15.91:0/3300056677 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4a9806aed0 0x7f4a9806d390 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4aa80011e0 tx=0x7f4aa8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:24.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:24.863+0000 7f4aa6ffd700 1 -- 172.21.15.91:0/3300056677 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4aac053ad0 con 0x7f4ab80b4150 2023-11-24T06:08:25.056 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.053+0000 7f4abc904700 1 -- 172.21.15.91:0/3300056677 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ab8147c10 con 0x7f4ab80b4150 2023-11-24T06:08:25.057 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.054+0000 7f4aa6ffd700 1 -- 172.21.15.91:0/3300056677 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7f4aac017020 con 0x7f4ab80b4150 2023-11-24T06:08:25.057 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:25.058 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":20,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:02.415612+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:25.060 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.056+0000 7f4abc904700 1 -- 172.21.15.91:0/3300056677 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4a9806aed0 msgr2=0x7f4a9806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.060 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.056+0000 7f4abc904700 1 --2- 172.21.15.91:0/3300056677 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4a9806aed0 0x7f4a9806d390 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4aa80011e0 tx=0x7f4aa8009040 comp rx=0 tx=0).stop 2023-11-24T06:08:25.060 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.056+0000 7f4abc904700 1 -- 172.21.15.91:0/3300056677 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ab80b4150 msgr2=0x7f4ab80ca630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.060 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.056+0000 7f4abc904700 1 --2- 172.21.15.91:0/3300056677 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ab80b4150 0x7f4ab80ca630 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4aac00c700 tx=0x7f4aac00ca10 comp rx=0 tx=0).stop 2023-11-24T06:08:25.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.056+0000 7f4abc904700 1 -- 172.21.15.91:0/3300056677 shutdown_connections 2023-11-24T06:08:25.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.056+0000 7f4abc904700 1 --2- 172.21.15.91:0/3300056677 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4a9806aed0 0x7f4a9806d390 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.057+0000 7f4abc904700 1 --2- 172.21.15.91:0/3300056677 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4ab806b620 0x7f4ab81e5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.057+0000 7f4abc904700 1 --2- 172.21.15.91:0/3300056677 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4ab80b5ab0 0x7f4ab80cab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.057+0000 7f4abc904700 1 --2- 172.21.15.91:0/3300056677 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4ab80b4150 0x7f4ab80ca630 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.057+0000 7f4abc904700 1 -- 172.21.15.91:0/3300056677 >> 172.21.15.91:0/3300056677 conn(0x7f4ab80b2740 msgr2=0x7f4ab81464d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:25.062 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.057+0000 7f4abc904700 1 -- 172.21.15.91:0/3300056677 shutdown_connections 2023-11-24T06:08:25.062 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.057+0000 7f4abc904700 1 -- 172.21.15.91:0/3300056677 wait complete. 2023-11-24T06:08:25.072 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:07:58.710106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:07:59.774819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:08:25.072 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:08:25.194 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.192+0000 7fdf80278700 1 Processor -- start 2023-11-24T06:08:25.195 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.192+0000 7fdf80278700 1 -- start start 2023-11-24T06:08:25.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf80278700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdf7815e1c0 0x7fdf7806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf80278700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdf780b0e90 0x7fdf7806f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf80278700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fdf7806fa80 0x7fdf78161d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf80278700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf7806ff10 con 0x7fdf7806fa80 2023-11-24T06:08:25.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf80278700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf780d1230 con 0x7fdf7815e1c0 2023-11-24T06:08:25.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf80278700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf780d0c70 con 0x7fdf780b0e90 2023-11-24T06:08:25.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf7e815700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fdf7806fa80 0x7fdf78161d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:25.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf7e815700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fdf7806fa80 0x7fdf78161d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58558/0 (socket says 172.21.15.91:58558) 2023-11-24T06:08:25.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf7e815700 1 -- 172.21.15.91:0/96466505 learned_addr learned my addr 172.21.15.91:0/96466505 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:25.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf7e014700 1 --2- 172.21.15.91:0/96466505 >> v2:172.21.15.94:3300/0 conn(0x7fdf7815e1c0 0x7fdf7806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:25.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf7d813700 1 --2- 172.21.15.91:0/96466505 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdf780b0e90 0x7fdf7806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:25.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf7e815700 1 -- 172.21.15.91:0/96466505 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdf780b0e90 msgr2=0x7fdf7806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:25.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf7e815700 1 --2- 172.21.15.91:0/96466505 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdf780b0e90 0x7fdf7806f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf7e815700 1 -- 172.21.15.91:0/96466505 >> v2:172.21.15.94:3300/0 conn(0x7fdf7815e1c0 msgr2=0x7fdf7806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:25.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf7e815700 1 --2- 172.21.15.91:0/96466505 >> v2:172.21.15.94:3300/0 conn(0x7fdf7815e1c0 0x7fdf7806eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf7e815700 1 -- 172.21.15.91:0/96466505 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf780d0f50 con 0x7fdf7806fa80 2023-11-24T06:08:25.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf7e014700 1 --2- 172.21.15.91:0/96466505 >> v2:172.21.15.94:3300/0 conn(0x7fdf7815e1c0 0x7fdf7806eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:25.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf7e815700 1 --2- 172.21.15.91:0/96466505 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdf7806fa80 0x7fdf78161d90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdf78071990 tx=0x7fdf7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=325fa97b0361a586 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:25.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.193+0000 7fdf7d012700 1 -- 172.21.15.91:0/96466505 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7000e070 con 0x7fdf7806fa80 2023-11-24T06:08:25.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.194+0000 7fdf7d012700 1 -- 172.21.15.91:0/96466505 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7000b910 con 0x7fdf7806fa80 2023-11-24T06:08:25.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.194+0000 7fdf7d012700 1 -- 172.21.15.91:0/96466505 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7000bcc0 con 0x7fdf7806fa80 2023-11-24T06:08:25.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.194+0000 7fdf80278700 1 -- 172.21.15.91:0/96466505 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdf7806fa80 msgr2=0x7fdf78161d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.194+0000 7fdf80278700 1 --2- 172.21.15.91:0/96466505 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdf7806fa80 0x7fdf78161d90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdf78071990 tx=0x7fdf7000b330 comp rx=0 tx=0).stop 2023-11-24T06:08:25.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.194+0000 7fdf80278700 1 -- 172.21.15.91:0/96466505 shutdown_connections 2023-11-24T06:08:25.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.194+0000 7fdf80278700 1 --2- 172.21.15.91:0/96466505 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdf7806fa80 0x7fdf78161d90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.194+0000 7fdf80278700 1 --2- 172.21.15.91:0/96466505 >> v2:172.21.15.94:3300/0 conn(0x7fdf7815e1c0 0x7fdf7806eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.194+0000 7fdf80278700 1 --2- 172.21.15.91:0/96466505 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdf780b0e90 0x7fdf7806f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.194+0000 7fdf80278700 1 -- 172.21.15.91:0/96466505 >> 172.21.15.91:0/96466505 conn(0x7fdf780b3720 msgr2=0x7fdf780b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:25.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.194+0000 7fdf80278700 1 -- 172.21.15.91:0/96466505 shutdown_connections 2023-11-24T06:08:25.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.194+0000 7fdf80278700 1 -- 172.21.15.91:0/96466505 wait complete. 2023-11-24T06:08:25.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.195+0000 7fdf80278700 1 Processor -- start 2023-11-24T06:08:25.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.195+0000 7fdf80278700 1 -- start start 2023-11-24T06:08:25.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.196+0000 7fdf80278700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdf7806fa80 0x7fdf780ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.196+0000 7fdf80278700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdf780b0e90 0x7fdf781f4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.196+0000 7fdf80278700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf7815e1c0 0x7fdf780cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.196+0000 7fdf80278700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf780c8660 con 0x7fdf780b0e90 2023-11-24T06:08:25.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.196+0000 7fdf80278700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf78160010 con 0x7fdf7815e1c0 2023-11-24T06:08:25.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.196+0000 7fdf80278700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf781f48b0 con 0x7fdf7806fa80 2023-11-24T06:08:25.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.196+0000 7fdf7d813700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdf780b0e90 0x7fdf781f4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:25.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.196+0000 7fdf7d813700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdf780b0e90 0x7fdf781f4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58562/0 (socket says 172.21.15.91:58562) 2023-11-24T06:08:25.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.196+0000 7fdf7d813700 1 -- 172.21.15.91:0/1650737609 learned_addr learned my addr 172.21.15.91:0/1650737609 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:25.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.196+0000 7fdf7e014700 1 --2- 172.21.15.91:0/1650737609 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdf7806fa80 0x7fdf780ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:25.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.196+0000 7fdf7e815700 1 --2- 172.21.15.91:0/1650737609 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf7815e1c0 0x7fdf780cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:25.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.197+0000 7fdf7d813700 1 -- 172.21.15.91:0/1650737609 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdf7806fa80 msgr2=0x7fdf780ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.197+0000 7fdf7d813700 1 --2- 172.21.15.91:0/1650737609 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdf7806fa80 0x7fdf780ccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.197+0000 7fdf7d813700 1 -- 172.21.15.91:0/1650737609 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf7815e1c0 msgr2=0x7fdf780cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.197+0000 7fdf7d813700 1 --2- 172.21.15.91:0/1650737609 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf7815e1c0 0x7fdf780cb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.197+0000 7fdf7d813700 1 -- 172.21.15.91:0/1650737609 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf70009b30 con 0x7fdf780b0e90 2023-11-24T06:08:25.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.197+0000 7fdf7d813700 1 --2- 172.21.15.91:0/1650737609 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdf780b0e90 0x7fdf781f4370 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800ea00 tx=0x7fdf6800ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:25.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.197+0000 7fdf6effd700 1 -- 172.21.15.91:0/1650737609 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf68009730 con 0x7fdf780b0e90 2023-11-24T06:08:25.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.197+0000 7fdf6effd700 1 -- 172.21.15.91:0/1650737609 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6800eea0 con 0x7fdf780b0e90 2023-11-24T06:08:25.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.197+0000 7fdf80278700 1 -- 172.21.15.91:0/1650737609 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7815fa00 con 0x7fdf780b0e90 2023-11-24T06:08:25.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.197+0000 7fdf6effd700 1 -- 172.21.15.91:0/1650737609 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6800ca70 con 0x7fdf780b0e90 2023-11-24T06:08:25.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.197+0000 7fdf80278700 1 -- 172.21.15.91:0/1650737609 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf780c9a00 con 0x7fdf780b0e90 2023-11-24T06:08:25.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.198+0000 7fdf80278700 1 -- 172.21.15.91:0/1650737609 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7815f860 con 0x7fdf780b0e90 2023-11-24T06:08:25.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.198+0000 7fdf6effd700 1 -- 172.21.15.91:0/1650737609 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdf6800cc10 con 0x7fdf780b0e90 2023-11-24T06:08:25.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.199+0000 7fdf6effd700 1 --2- 172.21.15.91:0/1650737609 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdf5c06aed0 0x7fdf5c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.199+0000 7fdf6effd700 1 -- 172.21.15.91:0/1650737609 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf68089600 con 0x7fdf780b0e90 2023-11-24T06:08:25.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.199+0000 7fdf7e014700 1 --2- 172.21.15.91:0/1650737609 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdf5c06aed0 0x7fdf5c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:25.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.199+0000 7fdf7e014700 1 --2- 172.21.15.91:0/1650737609 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdf5c06aed0 0x7fdf5c06d390 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdf74006f10 tx=0x7fdf74008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:25.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.203+0000 7fdf6effd700 1 -- 172.21.15.91:0/1650737609 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf7815f860 con 0x7fdf780b0e90 2023-11-24T06:08:25.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.392+0000 7fdf80278700 1 -- 172.21.15.91:0/1650737609 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdf781601c0 con 0x7fdf780b0e90 2023-11-24T06:08:25.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.392+0000 7fdf6effd700 1 -- 172.21.15.91:0/1650737609 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdf781601c0 con 0x7fdf780b0e90 2023-11-24T06:08:25.396 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:08:25.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.394+0000 7fdf80278700 1 -- 172.21.15.91:0/1650737609 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdf5c06aed0 msgr2=0x7fdf5c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.394+0000 7fdf80278700 1 --2- 172.21.15.91:0/1650737609 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdf5c06aed0 0x7fdf5c06d390 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdf74006f10 tx=0x7fdf74008040 comp rx=0 tx=0).stop 2023-11-24T06:08:25.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.394+0000 7fdf80278700 1 -- 172.21.15.91:0/1650737609 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdf780b0e90 msgr2=0x7fdf781f4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.394+0000 7fdf80278700 1 --2- 172.21.15.91:0/1650737609 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdf780b0e90 0x7fdf781f4370 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdf6800ea00 tx=0x7fdf6800ed10 comp rx=0 tx=0).stop 2023-11-24T06:08:25.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.395+0000 7fdf80278700 1 -- 172.21.15.91:0/1650737609 shutdown_connections 2023-11-24T06:08:25.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.395+0000 7fdf80278700 1 --2- 172.21.15.91:0/1650737609 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdf5c06aed0 0x7fdf5c06d390 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.395+0000 7fdf80278700 1 --2- 172.21.15.91:0/1650737609 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdf7815e1c0 0x7fdf780cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.395+0000 7fdf80278700 1 --2- 172.21.15.91:0/1650737609 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdf7806fa80 0x7fdf780ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.395+0000 7fdf80278700 1 --2- 172.21.15.91:0/1650737609 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdf780b0e90 0x7fdf781f4370 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.395+0000 7fdf80278700 1 -- 172.21.15.91:0/1650737609 >> 172.21.15.91:0/1650737609 conn(0x7fdf780b3720 msgr2=0x7fdf78070170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:25.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.395+0000 7fdf80278700 1 -- 172.21.15.91:0/1650737609 shutdown_connections 2023-11-24T06:08:25.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.395+0000 7fdf80278700 1 -- 172.21.15.91:0/1650737609 wait complete. 2023-11-24T06:08:25.409 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:08:25.534 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.531+0000 7f1a66081700 1 Processor -- start 2023-11-24T06:08:25.534 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.531+0000 7f1a66081700 1 -- start start 2023-11-24T06:08:25.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.532+0000 7f1a66081700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f1a6015bf10 0x7f1a60066ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.532+0000 7f1a66081700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a6006b7c0 0x7f1a6006bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.532+0000 7f1a66081700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a600b10c0 con 0x7f1a600af980 2023-11-24T06:08:25.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.532+0000 7f1a66081700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6006c190 con 0x7f1a6015bf10 2023-11-24T06:08:25.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.532+0000 7f1a66081700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a600af730 con 0x7f1a6006b7c0 2023-11-24T06:08:25.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.532+0000 7f1a5f7fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f1a6015bf10 0x7f1a60066ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:25.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.532+0000 7f1a5f7fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f1a6015bf10 0x7f1a60066ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58566/0 (socket says 172.21.15.91:58566) 2023-11-24T06:08:25.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.532+0000 7f1a5f7fe700 1 -- 172.21.15.91:0/1715021099 learned_addr learned my addr 172.21.15.91:0/1715021099 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:25.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.532+0000 7f1a5ffff700 1 --2- 172.21.15.91:0/1715021099 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a6006b7c0 0x7f1a6006bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:25.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.532+0000 7f1a5f7fe700 1 -- 172.21.15.91:0/1715021099 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a6006b7c0 msgr2=0x7f1a6006bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:25.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.532+0000 7f1a5f7fe700 1 --2- 172.21.15.91:0/1715021099 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a6006b7c0 0x7f1a6006bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.532+0000 7f1a5f7fe700 1 -- 172.21.15.91:0/1715021099 >> v1:172.21.15.94:6789/0 conn(0x7f1a600af980 legacy=0x7f1a600b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.532+0000 7f1a5f7fe700 1 -- 172.21.15.91:0/1715021099 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a600b1c60 con 0x7f1a6015bf10 2023-11-24T06:08:25.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.532+0000 7f1a5f7fe700 1 --2- 172.21.15.91:0/1715021099 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a6015bf10 0x7f1a60066ce0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1a540098f0 tx=0x7f1a54009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670563f689b7baac server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:25.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.533+0000 7f1a5e7fc700 1 -- 172.21.15.91:0/1715021099 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5400b8a0 con 0x7f1a6015bf10 2023-11-24T06:08:25.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.533+0000 7f1a5e7fc700 1 -- 172.21.15.91:0/1715021099 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5400ba00 con 0x7f1a6015bf10 2023-11-24T06:08:25.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.533+0000 7f1a5e7fc700 1 -- 172.21.15.91:0/1715021099 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5400bd90 con 0x7f1a6015bf10 2023-11-24T06:08:25.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.533+0000 7f1a66081700 1 -- 172.21.15.91:0/1715021099 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a6015bf10 msgr2=0x7f1a60066ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.533+0000 7f1a66081700 1 --2- 172.21.15.91:0/1715021099 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a6015bf10 0x7f1a60066ce0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1a540098f0 tx=0x7f1a54009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:25.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.533+0000 7f1a66081700 1 -- 172.21.15.91:0/1715021099 shutdown_connections 2023-11-24T06:08:25.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.533+0000 7f1a66081700 1 --2- 172.21.15.91:0/1715021099 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a6006b7c0 0x7f1a6006bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.533+0000 7f1a66081700 1 --2- 172.21.15.91:0/1715021099 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a6015bf10 0x7f1a60066ce0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.533+0000 7f1a66081700 1 -- 172.21.15.91:0/1715021099 >> 172.21.15.91:0/1715021099 conn(0x7f1a600b5070 msgr2=0x7f1a600b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:25.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.533+0000 7f1a66081700 1 -- 172.21.15.91:0/1715021099 shutdown_connections 2023-11-24T06:08:25.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.533+0000 7f1a66081700 1 -- 172.21.15.91:0/1715021099 wait complete. 2023-11-24T06:08:25.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.534+0000 7f1a66081700 1 Processor -- start 2023-11-24T06:08:25.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.535+0000 7f1a66081700 1 -- start start 2023-11-24T06:08:25.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.535+0000 7f1a66081700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a6006b7c0 0x7f1a60149d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.535+0000 7f1a66081700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a600af980 0x7f1a60147b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.535+0000 7f1a66081700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a6015bf10 0x7f1a601483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.535+0000 7f1a66081700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6014a290 con 0x7f1a600af980 2023-11-24T06:08:25.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.535+0000 7f1a66081700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a601480d0 con 0x7f1a6006b7c0 2023-11-24T06:08:25.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.535+0000 7f1a66081700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a60148cb0 con 0x7f1a6015bf10 2023-11-24T06:08:25.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.535+0000 7f1a5effd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a600af980 0x7f1a60147b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:25.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.535+0000 7f1a5effd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a600af980 0x7f1a60147b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58574/0 (socket says 172.21.15.91:58574) 2023-11-24T06:08:25.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.535+0000 7f1a5effd700 1 -- 172.21.15.91:0/3848156962 learned_addr learned my addr 172.21.15.91:0/3848156962 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:25.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.535+0000 7f1a5f7fe700 1 --2- 172.21.15.91:0/3848156962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a6006b7c0 0x7f1a60149d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:25.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.536+0000 7f1a5ffff700 1 --2- 172.21.15.91:0/3848156962 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a6015bf10 0x7f1a601483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:25.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.536+0000 7f1a5effd700 1 -- 172.21.15.91:0/3848156962 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a6015bf10 msgr2=0x7f1a601483f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.536+0000 7f1a5effd700 1 --2- 172.21.15.91:0/3848156962 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a6015bf10 0x7f1a601483f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.536+0000 7f1a5effd700 1 -- 172.21.15.91:0/3848156962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a6006b7c0 msgr2=0x7f1a60149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.536+0000 7f1a5effd700 1 --2- 172.21.15.91:0/3848156962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a6006b7c0 0x7f1a60149d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.536+0000 7f1a5effd700 1 -- 172.21.15.91:0/3848156962 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a54009550 con 0x7f1a600af980 2023-11-24T06:08:25.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.536+0000 7f1a5effd700 1 --2- 172.21.15.91:0/3848156962 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a600af980 0x7f1a60147b90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000c4e0 tx=0x7f1a5000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:25.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.536+0000 7f1a47fff700 1 -- 172.21.15.91:0/3848156962 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a500052d0 con 0x7f1a600af980 2023-11-24T06:08:25.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.536+0000 7f1a47fff700 1 -- 172.21.15.91:0/3848156962 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a50005430 con 0x7f1a600af980 2023-11-24T06:08:25.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.536+0000 7f1a66081700 1 -- 172.21.15.91:0/3848156962 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a60148f90 con 0x7f1a600af980 2023-11-24T06:08:25.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.536+0000 7f1a47fff700 1 -- 172.21.15.91:0/3848156962 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a500038c0 con 0x7f1a600af980 2023-11-24T06:08:25.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.536+0000 7f1a66081700 1 -- 172.21.15.91:0/3848156962 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a60149550 con 0x7f1a600af980 2023-11-24T06:08:25.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.537+0000 7f1a66081700 1 -- 172.21.15.91:0/3848156962 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a60149ba0 con 0x7f1a600af980 2023-11-24T06:08:25.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.537+0000 7f1a47fff700 1 -- 172.21.15.91:0/3848156962 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1a50003a20 con 0x7f1a600af980 2023-11-24T06:08:25.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.538+0000 7f1a47fff700 1 --2- 172.21.15.91:0/3848156962 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1a4c06aed0 0x7f1a4c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.538+0000 7f1a47fff700 1 -- 172.21.15.91:0/3848156962 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a50088880 con 0x7f1a600af980 2023-11-24T06:08:25.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.538+0000 7f1a5f7fe700 1 --2- 172.21.15.91:0/3848156962 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1a4c06aed0 0x7f1a4c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:25.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.538+0000 7f1a5f7fe700 1 --2- 172.21.15.91:0/3848156962 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1a4c06aed0 0x7f1a4c06d390 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1a54014040 tx=0x7f1a54008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:25.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.542+0000 7f1a47fff700 1 -- 172.21.15.91:0/3848156962 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a500539c0 con 0x7f1a600af980 2023-11-24T06:08:25.736 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.733+0000 7f1a66081700 1 -- 172.21.15.91:0/3848156962 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1a60075480 con 0x7f1a600af980 2023-11-24T06:08:25.737 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.734+0000 7f1a47fff700 1 -- 172.21.15.91:0/3848156962 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1a50017020 con 0x7f1a600af980 2023-11-24T06:08:25.737 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 32 2023-11-24T06:08:25.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.736+0000 7f1a66081700 1 -- 172.21.15.91:0/3848156962 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1a4c06aed0 msgr2=0x7f1a4c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.736+0000 7f1a66081700 1 --2- 172.21.15.91:0/3848156962 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1a4c06aed0 0x7f1a4c06d390 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1a54014040 tx=0x7f1a54008030 comp rx=0 tx=0).stop 2023-11-24T06:08:25.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.736+0000 7f1a66081700 1 -- 172.21.15.91:0/3848156962 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a600af980 msgr2=0x7f1a60147b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.736+0000 7f1a66081700 1 --2- 172.21.15.91:0/3848156962 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a600af980 0x7f1a60147b90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1a5000c4e0 tx=0x7f1a5000c7f0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.736+0000 7f1a66081700 1 -- 172.21.15.91:0/3848156962 shutdown_connections 2023-11-24T06:08:25.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.736+0000 7f1a66081700 1 --2- 172.21.15.91:0/3848156962 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1a4c06aed0 0x7f1a4c06d390 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.736+0000 7f1a66081700 1 --2- 172.21.15.91:0/3848156962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a6006b7c0 0x7f1a60149d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.736+0000 7f1a66081700 1 --2- 172.21.15.91:0/3848156962 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a6015bf10 0x7f1a601483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.736+0000 7f1a66081700 1 --2- 172.21.15.91:0/3848156962 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a600af980 0x7f1a60147b90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.736+0000 7f1a66081700 1 -- 172.21.15.91:0/3848156962 >> 172.21.15.91:0/3848156962 conn(0x7f1a600b5070 msgr2=0x7f1a6006d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:25.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.736+0000 7f1a66081700 1 -- 172.21.15.91:0/3848156962 shutdown_connections 2023-11-24T06:08:25.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.737+0000 7f1a66081700 1 -- 172.21.15.91:0/3848156962 wait complete. 2023-11-24T06:08:25.752 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:08:25.874 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.872+0000 7fd99ca40700 1 Processor -- start 2023-11-24T06:08:25.875 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.872+0000 7fd99ca40700 1 -- start start 2023-11-24T06:08:25.876 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.873+0000 7fd99ca40700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd998143ae0 0x7fd9980b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.876 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.873+0000 7fd99ca40700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd9981440f0 0x7fd998071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.876 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.873+0000 7fd99ca40700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd998143ec0 con 0x7fd9980720f0 2023-11-24T06:08:25.876 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.873+0000 7fd99ca40700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9980b40a0 con 0x7fd998143ae0 2023-11-24T06:08:25.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.873+0000 7fd99ca40700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd998072560 con 0x7fd9981440f0 2023-11-24T06:08:25.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.873+0000 7fd99659c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd998143ae0 0x7fd9980b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:25.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.873+0000 7fd996d9d700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fd9980720f0 0x7fd99815e8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53518/0 (socket says 172.21.15.91:53518) 2023-11-24T06:08:25.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.873+0000 7fd996d9d700 1 -- 172.21.15.91:0/3909830670 learned_addr learned my addr 172.21.15.91:0/3909830670 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:25.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.873+0000 7fd99659c700 1 -- 172.21.15.91:0/3909830670 >> v2:172.21.15.94:3300/0 conn(0x7fd9981440f0 msgr2=0x7fd998071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:25.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.873+0000 7fd99659c700 1 --2- 172.21.15.91:0/3909830670 >> v2:172.21.15.94:3300/0 conn(0x7fd9981440f0 0x7fd998071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.873+0000 7fd99659c700 1 -- 172.21.15.91:0/3909830670 >> v1:172.21.15.91:6789/0 conn(0x7fd9980720f0 legacy=0x7fd99815e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.873+0000 7fd99659c700 1 -- 172.21.15.91:0/3909830670 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9980b3ae0 con 0x7fd998143ae0 2023-11-24T06:08:25.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.873+0000 7fd99659c700 1 --2- 172.21.15.91:0/3909830670 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd998143ae0 0x7fd9980b51b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd9980d1b00 tx=0x7fd988009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d82a82fe3d58dcc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:25.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.873+0000 7fd99559a700 1 -- 172.21.15.91:0/3909830670 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd988011830 con 0x7fd998143ae0 2023-11-24T06:08:25.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.873+0000 7fd99559a700 1 -- 172.21.15.91:0/3909830670 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd988011990 con 0x7fd998143ae0 2023-11-24T06:08:25.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.874+0000 7fd99559a700 1 -- 172.21.15.91:0/3909830670 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd988011d20 con 0x7fd998143ae0 2023-11-24T06:08:25.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.874+0000 7fd99ca40700 1 -- 172.21.15.91:0/3909830670 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd998143ae0 msgr2=0x7fd9980b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.874+0000 7fd99ca40700 1 --2- 172.21.15.91:0/3909830670 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd998143ae0 0x7fd9980b51b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd9980d1b00 tx=0x7fd988009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:25.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.874+0000 7fd99ca40700 1 -- 172.21.15.91:0/3909830670 shutdown_connections 2023-11-24T06:08:25.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.874+0000 7fd99ca40700 1 --2- 172.21.15.91:0/3909830670 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd998143ae0 0x7fd9980b51b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.874+0000 7fd99ca40700 1 --2- 172.21.15.91:0/3909830670 >> v2:172.21.15.94:3300/0 conn(0x7fd9981440f0 0x7fd998071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.874+0000 7fd99ca40700 1 -- 172.21.15.91:0/3909830670 >> 172.21.15.91:0/3909830670 conn(0x7fd9980b2740 msgr2=0x7fd9980b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:25.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.874+0000 7fd99ca40700 1 -- 172.21.15.91:0/3909830670 shutdown_connections 2023-11-24T06:08:25.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.874+0000 7fd99ca40700 1 -- 172.21.15.91:0/3909830670 wait complete. 2023-11-24T06:08:25.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.875+0000 7fd99ca40700 1 Processor -- start 2023-11-24T06:08:25.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.875+0000 7fd99ca40700 1 -- start start 2023-11-24T06:08:25.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.876+0000 7fd99ca40700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd9980720f0 0x7fd99813d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.876+0000 7fd99ca40700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd998143ae0 0x7fd9981514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.876+0000 7fd99ca40700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd9981440f0 0x7fd9981f4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.876+0000 7fd99ca40700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9981401e0 con 0x7fd9980720f0 2023-11-24T06:08:25.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.876+0000 7fd99ca40700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd998151f80 con 0x7fd998143ae0 2023-11-24T06:08:25.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.876+0000 7fd99ca40700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd998152260 con 0x7fd9981440f0 2023-11-24T06:08:25.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.876+0000 7fd99659c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd9980720f0 0x7fd99813d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:25.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.876+0000 7fd99659c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd9980720f0 0x7fd99813d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58586/0 (socket says 172.21.15.91:58586) 2023-11-24T06:08:25.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.876+0000 7fd99659c700 1 -- 172.21.15.91:0/162993993 learned_addr learned my addr 172.21.15.91:0/162993993 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:25.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.877+0000 7fd99659c700 1 -- 172.21.15.91:0/162993993 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd9981440f0 msgr2=0x7fd9981f4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.877+0000 7fd99659c700 1 --2- 172.21.15.91:0/162993993 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd9981440f0 0x7fd9981f4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.877+0000 7fd99659c700 1 -- 172.21.15.91:0/162993993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd998143ae0 msgr2=0x7fd9981514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:25.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.877+0000 7fd99659c700 1 --2- 172.21.15.91:0/162993993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd998143ae0 0x7fd9981514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:25.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.877+0000 7fd99659c700 1 -- 172.21.15.91:0/162993993 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9880094a0 con 0x7fd9980720f0 2023-11-24T06:08:25.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.877+0000 7fd99659c700 1 --2- 172.21.15.91:0/162993993 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd9980720f0 0x7fd99813d1e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd988014040 tx=0x7fd988004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:25.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.877+0000 7fd986ffd700 1 -- 172.21.15.91:0/162993993 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9880045f0 con 0x7fd9980720f0 2023-11-24T06:08:25.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.877+0000 7fd986ffd700 1 -- 172.21.15.91:0/162993993 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd98801e070 con 0x7fd9980720f0 2023-11-24T06:08:25.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.877+0000 7fd986ffd700 1 -- 172.21.15.91:0/162993993 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd988003ae0 con 0x7fd9980720f0 2023-11-24T06:08:25.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.877+0000 7fd99ca40700 1 -- 172.21.15.91:0/162993993 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd99813fe00 con 0x7fd9980720f0 2023-11-24T06:08:25.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.877+0000 7fd99ca40700 1 -- 172.21.15.91:0/162993993 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd99813fa20 con 0x7fd9980720f0 2023-11-24T06:08:25.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.878+0000 7fd99ca40700 1 -- 172.21.15.91:0/162993993 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99813fc10 con 0x7fd9980720f0 2023-11-24T06:08:25.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.878+0000 7fd986ffd700 1 -- 172.21.15.91:0/162993993 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd988003c40 con 0x7fd9980720f0 2023-11-24T06:08:25.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.879+0000 7fd986ffd700 1 --2- 172.21.15.91:0/162993993 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd97806aed0 0x7fd97806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:25.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.879+0000 7fd995d9b700 1 --2- 172.21.15.91:0/162993993 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd97806aed0 0x7fd97806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:25.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.879+0000 7fd986ffd700 1 -- 172.21.15.91:0/162993993 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd98808a6d0 con 0x7fd9980720f0 2023-11-24T06:08:25.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.879+0000 7fd995d9b700 1 --2- 172.21.15.91:0/162993993 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd97806aed0 0x7fd97806d390 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd998152560 tx=0x7fd98c006c80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:25.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:25.883+0000 7fd986ffd700 1 -- 172.21.15.91:0/162993993 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd988055810 con 0x7fd9980720f0 2023-11-24T06:08:26.075 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.072+0000 7fd99ca40700 1 -- 172.21.15.91:0/162993993 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd998075480 con 0x7fd9980720f0 2023-11-24T06:08:26.076 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.073+0000 7fd986ffd700 1 -- 172.21.15.91:0/162993993 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd9880150a0 con 0x7fd9980720f0 2023-11-24T06:08:26.076 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 64 2023-11-24T06:08:26.078 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.075+0000 7fd99ca40700 1 -- 172.21.15.91:0/162993993 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd97806aed0 msgr2=0x7fd97806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.078 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.075+0000 7fd99ca40700 1 --2- 172.21.15.91:0/162993993 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd97806aed0 0x7fd97806d390 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd998152560 tx=0x7fd98c006c80 comp rx=0 tx=0).stop 2023-11-24T06:08:26.078 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.075+0000 7fd99ca40700 1 -- 172.21.15.91:0/162993993 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd9980720f0 msgr2=0x7fd99813d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.078 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.075+0000 7fd99ca40700 1 --2- 172.21.15.91:0/162993993 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd9980720f0 0x7fd99813d1e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd988014040 tx=0x7fd988004750 comp rx=0 tx=0).stop 2023-11-24T06:08:26.079 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.075+0000 7fd99ca40700 1 -- 172.21.15.91:0/162993993 shutdown_connections 2023-11-24T06:08:26.079 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.075+0000 7fd99ca40700 1 --2- 172.21.15.91:0/162993993 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd97806aed0 0x7fd97806d390 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.079 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.075+0000 7fd99ca40700 1 --2- 172.21.15.91:0/162993993 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd9981440f0 0x7fd9981f4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.079 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.075+0000 7fd99ca40700 1 --2- 172.21.15.91:0/162993993 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd9980720f0 0x7fd99813d1e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.079 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.075+0000 7fd99ca40700 1 --2- 172.21.15.91:0/162993993 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd998143ae0 0x7fd9981514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.079 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.075+0000 7fd99ca40700 1 -- 172.21.15.91:0/162993993 >> 172.21.15.91:0/162993993 conn(0x7fd9980b2740 msgr2=0x7fd99815a800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:26.079 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.075+0000 7fd99ca40700 1 -- 172.21.15.91:0/162993993 shutdown_connections 2023-11-24T06:08:26.079 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.075+0000 7fd99ca40700 1 -- 172.21.15.91:0/162993993 wait complete. 2023-11-24T06:08:26.090 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:26.209 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.207+0000 7fcb69b82700 1 Processor -- start 2023-11-24T06:08:26.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.207+0000 7fcb69b82700 1 -- start start 2023-11-24T06:08:26.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.207+0000 7fcb69b82700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcb6415bde0 0x7fcb6415c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.207+0000 7fcb69b82700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb640ce5c0 0x7fcb640cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.207+0000 7fcb69b82700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb640d0a90 con 0x7fcb640cf2b0 2023-11-24T06:08:26.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.207+0000 7fcb69b82700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb640cefb0 con 0x7fcb6415bde0 2023-11-24T06:08:26.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.207+0000 7fcb69b82700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6406ee10 con 0x7fcb640ce5c0 2023-11-24T06:08:26.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.208+0000 7fcb63fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb640ce5c0 0x7fcb640cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.208+0000 7fcb63fff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb640ce5c0 0x7fcb640cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:42030/0 (socket says 172.21.15.91:42030) 2023-11-24T06:08:26.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.208+0000 7fcb63fff700 1 -- 172.21.15.91:0/2744431994 learned_addr learned my addr 172.21.15.91:0/2744431994 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:26.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.208+0000 7fcb637fe700 1 --2- 172.21.15.91:0/2744431994 >> v2:172.21.15.94:3300/0 conn(0x7fcb6415bde0 0x7fcb6415c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.208+0000 7fcb63fff700 1 -- 172.21.15.91:0/2744431994 >> v2:172.21.15.94:3300/0 conn(0x7fcb6415bde0 msgr2=0x7fcb6415c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:26.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.208+0000 7fcb63fff700 1 --2- 172.21.15.91:0/2744431994 >> v2:172.21.15.94:3300/0 conn(0x7fcb6415bde0 0x7fcb6415c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.208+0000 7fcb63fff700 1 -- 172.21.15.91:0/2744431994 >> v1:172.21.15.94:6789/0 conn(0x7fcb640cf2b0 legacy=0x7fcb640d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.208+0000 7fcb63fff700 1 -- 172.21.15.91:0/2744431994 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6406f0f0 con 0x7fcb640ce5c0 2023-11-24T06:08:26.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.208+0000 7fcb627fc700 1 -- 172.21.15.91:0/2744431994 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311066634 0 0) 0x7fcb640d0a90 con 0x7fcb640cf2b0 2023-11-24T06:08:26.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.208+0000 7fcb637fe700 1 --2- 172.21.15.91:0/2744431994 >> v2:172.21.15.94:3300/0 conn(0x7fcb6415bde0 0x7fcb6415c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:08:26.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.208+0000 7fcb63fff700 1 --2- 172.21.15.91:0/2744431994 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb640ce5c0 0x7fcb640cea70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009e50 tx=0x7fcb5c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b1b48084bbf896a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:26.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.209+0000 7fcb627fc700 1 -- 172.21.15.91:0/2744431994 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c00e070 con 0x7fcb640ce5c0 2023-11-24T06:08:26.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.209+0000 7fcb627fc700 1 -- 172.21.15.91:0/2744431994 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c00b910 con 0x7fcb640ce5c0 2023-11-24T06:08:26.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.209+0000 7fcb627fc700 1 -- 172.21.15.91:0/2744431994 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c00bcc0 con 0x7fcb640ce5c0 2023-11-24T06:08:26.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.209+0000 7fcb69b82700 1 -- 172.21.15.91:0/2744431994 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb640ce5c0 msgr2=0x7fcb640cea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.209+0000 7fcb69b82700 1 --2- 172.21.15.91:0/2744431994 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb640ce5c0 0x7fcb640cea70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009e50 tx=0x7fcb5c00b330 comp rx=0 tx=0).stop 2023-11-24T06:08:26.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.209+0000 7fcb69b82700 1 -- 172.21.15.91:0/2744431994 shutdown_connections 2023-11-24T06:08:26.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.209+0000 7fcb69b82700 1 --2- 172.21.15.91:0/2744431994 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb640ce5c0 0x7fcb640cea70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.209+0000 7fcb69b82700 1 --2- 172.21.15.91:0/2744431994 >> v2:172.21.15.94:3300/0 conn(0x7fcb6415bde0 0x7fcb6415c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.209+0000 7fcb69b82700 1 -- 172.21.15.91:0/2744431994 >> 172.21.15.91:0/2744431994 conn(0x7fcb640b3720 msgr2=0x7fcb640b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:26.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.209+0000 7fcb69b82700 1 -- 172.21.15.91:0/2744431994 shutdown_connections 2023-11-24T06:08:26.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.209+0000 7fcb69b82700 1 -- 172.21.15.91:0/2744431994 wait complete. 2023-11-24T06:08:26.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.210+0000 7fcb69b82700 1 Processor -- start 2023-11-24T06:08:26.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.211+0000 7fcb69b82700 1 -- start start 2023-11-24T06:08:26.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.211+0000 7fcb69b82700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb640ce5c0 0x7fcb640c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.211+0000 7fcb69b82700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb640cf2b0 0x7fcb6415a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.211+0000 7fcb69b82700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcb6415bde0 0x7fcb6406d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.211+0000 7fcb69b82700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb640c8ca0 con 0x7fcb6415bde0 2023-11-24T06:08:26.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.211+0000 7fcb69b82700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb640c9220 con 0x7fcb640cf2b0 2023-11-24T06:08:26.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.211+0000 7fcb69b82700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6406dc80 con 0x7fcb640ce5c0 2023-11-24T06:08:26.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.211+0000 7fcb63fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcb6415bde0 0x7fcb6406d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.212+0000 7fcb63fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcb6415bde0 0x7fcb6406d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58604/0 (socket says 172.21.15.91:58604) 2023-11-24T06:08:26.220 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.212+0000 7fcb63fff700 1 -- 172.21.15.91:0/3725127494 learned_addr learned my addr 172.21.15.91:0/3725127494 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:26.220 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.212+0000 7fcb62ffd700 1 --2- 172.21.15.91:0/3725127494 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb640cf2b0 0x7fcb6415a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.220 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.212+0000 7fcb637fe700 1 --2- 172.21.15.91:0/3725127494 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb640ce5c0 0x7fcb640c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.220 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.212+0000 7fcb63fff700 1 -- 172.21.15.91:0/3725127494 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb640ce5c0 msgr2=0x7fcb640c49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.220 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.212+0000 7fcb63fff700 1 --2- 172.21.15.91:0/3725127494 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb640ce5c0 0x7fcb640c49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.220 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.212+0000 7fcb63fff700 1 -- 172.21.15.91:0/3725127494 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb640cf2b0 msgr2=0x7fcb6415a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.220 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.212+0000 7fcb63fff700 1 --2- 172.21.15.91:0/3725127494 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb640cf2b0 0x7fcb6415a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.220 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.212+0000 7fcb63fff700 1 -- 172.21.15.91:0/3725127494 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c0094a0 con 0x7fcb6415bde0 2023-11-24T06:08:26.220 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.212+0000 7fcb63fff700 1 --2- 172.21.15.91:0/3725127494 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcb6415bde0 0x7fcb6406d430 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcb6406ef50 tx=0x7fcb5c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:26.220 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.212+0000 7fcb5bfff700 1 -- 172.21.15.91:0/3725127494 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c003ea0 con 0x7fcb6415bde0 2023-11-24T06:08:26.220 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.212+0000 7fcb5bfff700 1 -- 172.21.15.91:0/3725127494 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c0048e0 con 0x7fcb6415bde0 2023-11-24T06:08:26.220 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.212+0000 7fcb5bfff700 1 -- 172.21.15.91:0/3725127494 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c01c680 con 0x7fcb6415bde0 2023-11-24T06:08:26.220 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.212+0000 7fcb69b82700 1 -- 172.21.15.91:0/3725127494 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5c009b30 con 0x7fcb6415bde0 2023-11-24T06:08:26.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.212+0000 7fcb69b82700 1 -- 172.21.15.91:0/3725127494 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb640c8eb0 con 0x7fcb6415bde0 2023-11-24T06:08:26.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.213+0000 7fcb69b82700 1 -- 172.21.15.91:0/3725127494 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb640c9070 con 0x7fcb6415bde0 2023-11-24T06:08:26.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.213+0000 7fcb5bfff700 1 -- 172.21.15.91:0/3725127494 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcb5c01c7e0 con 0x7fcb6415bde0 2023-11-24T06:08:26.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.214+0000 7fcb5bfff700 1 --2- 172.21.15.91:0/3725127494 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcb4406aed0 0x7fcb4406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.214+0000 7fcb5bfff700 1 -- 172.21.15.91:0/3725127494 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb5c08b150 con 0x7fcb6415bde0 2023-11-24T06:08:26.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.214+0000 7fcb637fe700 1 --2- 172.21.15.91:0/3725127494 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcb4406aed0 0x7fcb4406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.214+0000 7fcb637fe700 1 --2- 172.21.15.91:0/3725127494 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcb4406aed0 0x7fcb4406d390 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009cd0 tx=0x7fcb4c009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:26.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.219+0000 7fcb5bfff700 1 -- 172.21.15.91:0/3725127494 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb5c056290 con 0x7fcb6415bde0 2023-11-24T06:08:26.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.411+0000 7fcb69b82700 1 -- 172.21.15.91:0/3725127494 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb640c9390 con 0x7fcb6415bde0 2023-11-24T06:08:26.414 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.412+0000 7fcb5bfff700 1 -- 172.21.15.91:0/3725127494 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7fcb5c0598b0 con 0x7fcb6415bde0 2023-11-24T06:08:26.415 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:26.415 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":20,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:02.415612+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:26.417 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.414+0000 7fcb69b82700 1 -- 172.21.15.91:0/3725127494 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcb4406aed0 msgr2=0x7fcb4406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.417 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.414+0000 7fcb69b82700 1 --2- 172.21.15.91:0/3725127494 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcb4406aed0 0x7fcb4406d390 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009cd0 tx=0x7fcb4c009210 comp rx=0 tx=0).stop 2023-11-24T06:08:26.417 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.414+0000 7fcb69b82700 1 -- 172.21.15.91:0/3725127494 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcb6415bde0 msgr2=0x7fcb6406d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.417 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.414+0000 7fcb69b82700 1 --2- 172.21.15.91:0/3725127494 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcb6415bde0 0x7fcb6406d430 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcb6406ef50 tx=0x7fcb5c004ad0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.417 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.414+0000 7fcb69b82700 1 -- 172.21.15.91:0/3725127494 shutdown_connections 2023-11-24T06:08:26.417 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.414+0000 7fcb69b82700 1 --2- 172.21.15.91:0/3725127494 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcb4406aed0 0x7fcb4406d390 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.414+0000 7fcb69b82700 1 --2- 172.21.15.91:0/3725127494 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcb6415bde0 0x7fcb6406d430 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.414+0000 7fcb69b82700 1 --2- 172.21.15.91:0/3725127494 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcb640ce5c0 0x7fcb640c49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.414+0000 7fcb69b82700 1 --2- 172.21.15.91:0/3725127494 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcb640cf2b0 0x7fcb6415a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.414+0000 7fcb69b82700 1 -- 172.21.15.91:0/3725127494 >> 172.21.15.91:0/3725127494 conn(0x7fcb640b3720 msgr2=0x7fcb640b04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:26.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.414+0000 7fcb69b82700 1 -- 172.21.15.91:0/3725127494 shutdown_connections 2023-11-24T06:08:26.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.414+0000 7fcb69b82700 1 -- 172.21.15.91:0/3725127494 wait complete. 2023-11-24T06:08:26.429 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:07:58.710106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:07:59.774819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:08:26.429 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:08:26.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.549+0000 7f18e01f0700 1 Processor -- start 2023-11-24T06:08:26.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.549+0000 7f18e01f0700 1 -- start start 2023-11-24T06:08:26.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.550+0000 7f18e01f0700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f18d80b5e40 0x7f18d815ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.550+0000 7f18e01f0700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d806bb80 con 0x7f18d815e1e0 2023-11-24T06:08:26.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.550+0000 7f18e01f0700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d806be20 con 0x7f18d806b400 2023-11-24T06:08:26.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.550+0000 7f18e01f0700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d806c100 con 0x7f18d80b5e40 2023-11-24T06:08:26.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.550+0000 7f18ddf8c700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f18d815e1e0 0x7f18d80b12a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53542/0 (socket says 172.21.15.91:53542) 2023-11-24T06:08:26.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.550+0000 7f18ddf8c700 1 -- 172.21.15.91:0/887036663 learned_addr learned my addr 172.21.15.91:0/887036663 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:26.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18de78d700 1 --2- 172.21.15.91:0/887036663 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f18d80b5e40 0x7f18d815ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 -- 172.21.15.91:0/887036663 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503847022 0 0) 0x7f18d806bb80 con 0x7f18d815e1e0 2023-11-24T06:08:26.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 -- 172.21.15.91:0/887036663 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c40035b0 con 0x7f18d815e1e0 2023-11-24T06:08:26.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 -- 172.21.15.91:0/887036663 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370302568 0 0) 0x7f18d806be20 con 0x7f18d806b400 2023-11-24T06:08:26.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 -- 172.21.15.91:0/887036663 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18d806bb80 con 0x7f18d806b400 2023-11-24T06:08:26.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 -- 172.21.15.91:0/887036663 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1782500869 0 0) 0x7f18c40035b0 con 0x7f18d815e1e0 2023-11-24T06:08:26.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 -- 172.21.15.91:0/887036663 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f18d80b5e40 msgr2=0x7f18d815ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:26.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 --2- 172.21.15.91:0/887036663 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f18d80b5e40 0x7f18d815ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 -- 172.21.15.91:0/887036663 >> v1:172.21.15.94:6789/0 conn(0x7f18d806b400 legacy=0x7f18d815e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 -- 172.21.15.91:0/887036663 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d806c3e0 con 0x7f18d815e1e0 2023-11-24T06:08:26.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 -- 172.21.15.91:0/887036663 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f18c8001670 con 0x7f18d815e1e0 2023-11-24T06:08:26.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 -- 172.21.15.91:0/887036663 >> v1:172.21.15.91:6789/0 conn(0x7f18d815e1e0 legacy=0x7f18d80b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 --2- 172.21.15.91:0/887036663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18c4003bc0 0x7f18c4003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 --2- 172.21.15.91:0/887036663 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f18c4004850 0x7f18c4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 --2- 172.21.15.91:0/887036663 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f18c4005210 0x7f18c400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 -- 172.21.15.91:0/887036663 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d806c100 con 0x7f18c4005210 2023-11-24T06:08:26.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 -- 172.21.15.91:0/887036663 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c400c000 con 0x7f18c4003bc0 2023-11-24T06:08:26.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dcf8a700 1 -- 172.21.15.91:0/887036663 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c400c1c0 con 0x7f18c4004850 2023-11-24T06:08:26.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18de78d700 1 --2- 172.21.15.91:0/887036663 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f18d80b5e40 0x7f18d815ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:08:26.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18dd78b700 1 --2- 172.21.15.91:0/887036663 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f18c4004850 0x7f18c4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18ddf8c700 1 --2- 172.21.15.91:0/887036663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18c4003bc0 0x7f18c4003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18de78d700 1 --2- 172.21.15.91:0/887036663 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f18c4005210 0x7f18c400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18de78d700 1 -- 172.21.15.91:0/887036663 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f18c4004850 msgr2=0x7f18c4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:26.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18de78d700 1 --2- 172.21.15.91:0/887036663 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f18c4004850 0x7f18c4004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18de78d700 1 -- 172.21.15.91:0/887036663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18c4003bc0 msgr2=0x7f18c4003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:26.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18de78d700 1 --2- 172.21.15.91:0/887036663 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18c4003bc0 0x7f18c4003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.551+0000 7f18de78d700 1 -- 172.21.15.91:0/887036663 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c40035b0 con 0x7f18c4005210 2023-11-24T06:08:26.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.552+0000 7f18de78d700 1 --2- 172.21.15.91:0/887036663 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f18c4005210 0x7f18c400ba10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f18d80b6220 tx=0x7f18c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8863124c1f028569 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:26.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.552+0000 7f18dcf8a700 1 -- 172.21.15.91:0/887036663 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d0003c20 con 0x7f18c4005210 2023-11-24T06:08:26.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.552+0000 7f18dcf8a700 1 -- 172.21.15.91:0/887036663 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d000f040 con 0x7f18c4005210 2023-11-24T06:08:26.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.552+0000 7f18dcf8a700 1 -- 172.21.15.91:0/887036663 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d000ea60 con 0x7f18c4005210 2023-11-24T06:08:26.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.552+0000 7f18e01f0700 1 -- 172.21.15.91:0/887036663 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f18c4005210 msgr2=0x7f18c400ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.552+0000 7f18e01f0700 1 --2- 172.21.15.91:0/887036663 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f18c4005210 0x7f18c400ba10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f18d80b6220 tx=0x7f18c4005880 comp rx=0 tx=0).stop 2023-11-24T06:08:26.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.552+0000 7f18de78d700 1 -- 172.21.15.91:0/887036663 reap_dead start 2023-11-24T06:08:26.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.552+0000 7f18e01f0700 1 -- 172.21.15.91:0/887036663 shutdown_connections 2023-11-24T06:08:26.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.552+0000 7f18e01f0700 1 -- 172.21.15.91:0/887036663 >> 172.21.15.91:0/887036663 conn(0x7f18d80b5070 msgr2=0x7f18d80b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:26.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.552+0000 7f18e01f0700 1 -- 172.21.15.91:0/887036663 shutdown_connections 2023-11-24T06:08:26.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.552+0000 7f18e01f0700 1 -- 172.21.15.91:0/887036663 wait complete. 2023-11-24T06:08:26.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.553+0000 7f18e01f0700 1 Processor -- start 2023-11-24T06:08:26.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.553+0000 7f18e01f0700 1 -- start start 2023-11-24T06:08:26.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.554+0000 7f18e01f0700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f18c4005210 0x7f18d80cf030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.554+0000 7f18e01f0700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f18d815a670 0x7f18d80c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.554+0000 7f18e01f0700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18d80c9e60 0x7f18d80ae750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.554+0000 7f18e01f0700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d814b200 con 0x7f18c4005210 2023-11-24T06:08:26.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.554+0000 7f18e01f0700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d809af70 con 0x7f18d80c9e60 2023-11-24T06:08:26.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.554+0000 7f18e01f0700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d809a3a0 con 0x7f18d815a670 2023-11-24T06:08:26.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.554+0000 7f18ddf8c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f18c4005210 0x7f18d80cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.554+0000 7f18ddf8c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f18c4005210 0x7f18d80cf030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58620/0 (socket says 172.21.15.91:58620) 2023-11-24T06:08:26.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.554+0000 7f18ddf8c700 1 -- 172.21.15.91:0/3842528543 learned_addr learned my addr 172.21.15.91:0/3842528543 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:26.562 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.555+0000 7f18de78d700 1 --2- 172.21.15.91:0/3842528543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18d80c9e60 0x7f18d80ae750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.555+0000 7f18ddf8c700 1 -- 172.21.15.91:0/3842528543 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f18d815a670 msgr2=0x7f18d80c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.555+0000 7f18dd78b700 1 --2- 172.21.15.91:0/3842528543 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f18d815a670 0x7f18d80c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.555+0000 7f18ddf8c700 1 --2- 172.21.15.91:0/3842528543 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f18d815a670 0x7f18d80c9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.555+0000 7f18ddf8c700 1 -- 172.21.15.91:0/3842528543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18d80c9e60 msgr2=0x7f18d80ae750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.555+0000 7f18ddf8c700 1 --2- 172.21.15.91:0/3842528543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18d80c9e60 0x7f18d80ae750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.555+0000 7f18ddf8c700 1 -- 172.21.15.91:0/3842528543 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d809a640 con 0x7f18c4005210 2023-11-24T06:08:26.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.555+0000 7f18ddf8c700 1 --2- 172.21.15.91:0/3842528543 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f18c4005210 0x7f18d80cf030 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f18c8007370 tx=0x7f18c8007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:26.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.555+0000 7f18ce7fc700 1 -- 172.21.15.91:0/3842528543 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c80077a0 con 0x7f18c4005210 2023-11-24T06:08:26.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.555+0000 7f18ce7fc700 1 -- 172.21.15.91:0/3842528543 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c800f870 con 0x7f18c4005210 2023-11-24T06:08:26.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.555+0000 7f18e01f0700 1 -- 172.21.15.91:0/3842528543 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d809a920 con 0x7f18c4005210 2023-11-24T06:08:26.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.555+0000 7f18ce7fc700 1 -- 172.21.15.91:0/3842528543 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c800fc20 con 0x7f18c4005210 2023-11-24T06:08:26.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.555+0000 7f18e01f0700 1 -- 172.21.15.91:0/3842528543 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d80aec90 con 0x7f18c4005210 2023-11-24T06:08:26.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.556+0000 7f18e01f0700 1 -- 172.21.15.91:0/3842528543 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d8075480 con 0x7f18c4005210 2023-11-24T06:08:26.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.556+0000 7f18ce7fc700 1 -- 172.21.15.91:0/3842528543 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f18c800fd80 con 0x7f18c4005210 2023-11-24T06:08:26.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.557+0000 7f18ce7fc700 1 --2- 172.21.15.91:0/3842528543 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f18bc06aed0 0x7f18bc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.557+0000 7f18ce7fc700 1 -- 172.21.15.91:0/3842528543 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18c8089580 con 0x7f18c4005210 2023-11-24T06:08:26.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.557+0000 7f18dd78b700 1 --2- 172.21.15.91:0/3842528543 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f18bc06aed0 0x7f18bc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.557+0000 7f18dd78b700 1 --2- 172.21.15.91:0/3842528543 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f18bc06aed0 0x7f18bc06d390 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f18d8160e10 tx=0x7f18d4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:26.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.561+0000 7f18ce7fc700 1 -- 172.21.15.91:0/3842528543 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f18c80546c0 con 0x7f18c4005210 2023-11-24T06:08:26.755 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.752+0000 7f18e01f0700 1 -- 172.21.15.91:0/3842528543 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f18d80b5810 con 0x7f18c4005210 2023-11-24T06:08:26.755 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.753+0000 7f18ce7fc700 1 -- 172.21.15.91:0/3842528543 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f18c8057ce0 con 0x7f18c4005210 2023-11-24T06:08:26.756 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:08:26.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.755+0000 7f18e01f0700 1 -- 172.21.15.91:0/3842528543 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f18bc06aed0 msgr2=0x7f18bc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.755+0000 7f18e01f0700 1 --2- 172.21.15.91:0/3842528543 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f18bc06aed0 0x7f18bc06d390 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f18d8160e10 tx=0x7f18d4009040 comp rx=0 tx=0).stop 2023-11-24T06:08:26.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.755+0000 7f18e01f0700 1 -- 172.21.15.91:0/3842528543 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f18c4005210 msgr2=0x7f18d80cf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.755+0000 7f18e01f0700 1 --2- 172.21.15.91:0/3842528543 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f18c4005210 0x7f18d80cf030 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f18c8007370 tx=0x7f18c8007680 comp rx=0 tx=0).stop 2023-11-24T06:08:26.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.755+0000 7f18e01f0700 1 -- 172.21.15.91:0/3842528543 shutdown_connections 2023-11-24T06:08:26.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.755+0000 7f18e01f0700 1 --2- 172.21.15.91:0/3842528543 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f18bc06aed0 0x7f18bc06d390 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.755+0000 7f18e01f0700 1 --2- 172.21.15.91:0/3842528543 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f18d80c9e60 0x7f18d80ae750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.755+0000 7f18e01f0700 1 --2- 172.21.15.91:0/3842528543 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f18c4005210 0x7f18d80cf030 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.755+0000 7f18e01f0700 1 --2- 172.21.15.91:0/3842528543 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f18d815a670 0x7f18d80c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.755+0000 7f18e01f0700 1 -- 172.21.15.91:0/3842528543 >> 172.21.15.91:0/3842528543 conn(0x7f18d80b5070 msgr2=0x7f18d815fc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:26.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.755+0000 7f18e01f0700 1 -- 172.21.15.91:0/3842528543 shutdown_connections 2023-11-24T06:08:26.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.755+0000 7f18e01f0700 1 -- 172.21.15.91:0/3842528543 wait complete. 2023-11-24T06:08:26.769 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:08:26.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.889+0000 7feda5b4b700 1 Processor -- start 2023-11-24T06:08:26.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.889+0000 7feda5b4b700 1 -- start start 2023-11-24T06:08:26.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.890+0000 7feda5b4b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feda006b620 0x7feda00b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.890+0000 7feda5b4b700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7feda00b5ab0 0x7feda00b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.890+0000 7feda5b4b700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda0071f50 con 0x7feda01478c0 2023-11-24T06:08:26.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.890+0000 7feda5b4b700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feda0072230 con 0x7feda00b5ab0 2023-11-24T06:08:26.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.890+0000 7feda5b4b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda0072510 con 0x7feda006b620 2023-11-24T06:08:26.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.890+0000 7fed96ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7feda00b5ab0 0x7feda00b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.890+0000 7fed96ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7feda00b5ab0 0x7feda00b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58628/0 (socket says 172.21.15.91:58628) 2023-11-24T06:08:26.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.890+0000 7fed96ffd700 1 -- 172.21.15.91:0/3027895285 learned_addr learned my addr 172.21.15.91:0/3027895285 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:26.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.890+0000 7fed9f7fe700 1 --2- 172.21.15.91:0/3027895285 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feda006b620 0x7feda00b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.890+0000 7fed96ffd700 1 -- 172.21.15.91:0/3027895285 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feda006b620 msgr2=0x7feda00b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:26.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.890+0000 7fed96ffd700 1 --2- 172.21.15.91:0/3027895285 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feda006b620 0x7feda00b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.890+0000 7fed96ffd700 1 -- 172.21.15.91:0/3027895285 >> v1:172.21.15.94:6789/0 conn(0x7feda01478c0 legacy=0x7feda0161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.890+0000 7fed96ffd700 1 -- 172.21.15.91:0/3027895285 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda00727f0 con 0x7feda00b5ab0 2023-11-24T06:08:26.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.890+0000 7fed96ffd700 1 --2- 172.21.15.91:0/3027895285 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feda00b5ab0 0x7feda00b3f80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fed880098f0 tx=0x7fed88009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=842c9a9d88059d26 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:26.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.891+0000 7fed9effd700 1 -- 172.21.15.91:0/3027895285 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8800b8a0 con 0x7feda00b5ab0 2023-11-24T06:08:26.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.891+0000 7fed9effd700 1 -- 172.21.15.91:0/3027895285 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8800ba00 con 0x7feda00b5ab0 2023-11-24T06:08:26.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.891+0000 7fed9effd700 1 -- 172.21.15.91:0/3027895285 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8800bd90 con 0x7feda00b5ab0 2023-11-24T06:08:26.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.891+0000 7feda5b4b700 1 -- 172.21.15.91:0/3027895285 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feda00b5ab0 msgr2=0x7feda00b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.891+0000 7feda5b4b700 1 --2- 172.21.15.91:0/3027895285 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feda00b5ab0 0x7feda00b3f80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fed880098f0 tx=0x7fed88009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:26.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.891+0000 7feda5b4b700 1 -- 172.21.15.91:0/3027895285 shutdown_connections 2023-11-24T06:08:26.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.891+0000 7feda5b4b700 1 --2- 172.21.15.91:0/3027895285 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feda006b620 0x7feda00b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.891+0000 7feda5b4b700 1 --2- 172.21.15.91:0/3027895285 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feda00b5ab0 0x7feda00b3f80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.891+0000 7feda5b4b700 1 -- 172.21.15.91:0/3027895285 >> 172.21.15.91:0/3027895285 conn(0x7feda00b2740 msgr2=0x7feda00b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:26.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.891+0000 7feda5b4b700 1 -- 172.21.15.91:0/3027895285 shutdown_connections 2023-11-24T06:08:26.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.891+0000 7feda5b4b700 1 -- 172.21.15.91:0/3027895285 wait complete. 2023-11-24T06:08:26.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.892+0000 7feda5b4b700 1 Processor -- start 2023-11-24T06:08:26.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.893+0000 7feda5b4b700 1 -- start start 2023-11-24T06:08:26.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.893+0000 7feda5b4b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feda006b620 0x7feda01e1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.893+0000 7feda5b4b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feda00b5ab0 0x7feda01e4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.893+0000 7feda5b4b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feda01478c0 0x7feda01e4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.893+0000 7feda5b4b700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda01dcae0 con 0x7feda01478c0 2023-11-24T06:08:26.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.893+0000 7feda5b4b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda01e26e0 con 0x7feda006b620 2023-11-24T06:08:26.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.893+0000 7feda5b4b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda01e29c0 con 0x7feda00b5ab0 2023-11-24T06:08:26.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.893+0000 7fed9ffff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feda01478c0 0x7feda01e4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.893+0000 7fed9ffff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feda01478c0 0x7feda01e4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58636/0 (socket says 172.21.15.91:58636) 2023-11-24T06:08:26.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.893+0000 7fed9ffff700 1 -- 172.21.15.91:0/2423287118 learned_addr learned my addr 172.21.15.91:0/2423287118 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:26.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.894+0000 7fed9f7fe700 1 --2- 172.21.15.91:0/2423287118 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feda006b620 0x7feda01e1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.894+0000 7fed96ffd700 1 --2- 172.21.15.91:0/2423287118 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feda00b5ab0 0x7feda01e4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.894+0000 7fed9ffff700 1 -- 172.21.15.91:0/2423287118 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feda00b5ab0 msgr2=0x7feda01e4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.894+0000 7fed9ffff700 1 --2- 172.21.15.91:0/2423287118 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feda00b5ab0 0x7feda01e4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.894+0000 7fed9ffff700 1 -- 172.21.15.91:0/2423287118 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feda006b620 msgr2=0x7feda01e1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:26.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.894+0000 7fed9ffff700 1 --2- 172.21.15.91:0/2423287118 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feda006b620 0x7feda01e1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:26.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.894+0000 7fed9ffff700 1 -- 172.21.15.91:0/2423287118 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed88009550 con 0x7feda01478c0 2023-11-24T06:08:26.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.894+0000 7fed96ffd700 1 --2- 172.21.15.91:0/2423287118 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feda00b5ab0 0x7feda01e4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:26.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.894+0000 7fed9ffff700 1 --2- 172.21.15.91:0/2423287118 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feda01478c0 0x7feda01e4b70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fed9800cb50 tx=0x7fed9800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:26.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.894+0000 7fed9cff9700 1 -- 172.21.15.91:0/2423287118 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed98003c00 con 0x7feda01478c0 2023-11-24T06:08:26.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.894+0000 7fed9cff9700 1 -- 172.21.15.91:0/2423287118 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed98003d60 con 0x7feda01478c0 2023-11-24T06:08:26.901 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.894+0000 7feda5b4b700 1 -- 172.21.15.91:0/2423287118 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda01e2ca0 con 0x7feda01478c0 2023-11-24T06:08:26.901 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.894+0000 7fed9cff9700 1 -- 172.21.15.91:0/2423287118 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9800aac0 con 0x7feda01478c0 2023-11-24T06:08:26.901 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.894+0000 7feda5b4b700 1 -- 172.21.15.91:0/2423287118 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda01e23d0 con 0x7feda01478c0 2023-11-24T06:08:26.901 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.895+0000 7feda5b4b700 1 -- 172.21.15.91:0/2423287118 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda01dcc50 con 0x7feda01478c0 2023-11-24T06:08:26.901 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.895+0000 7fed9cff9700 1 -- 172.21.15.91:0/2423287118 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fed9800ac20 con 0x7feda01478c0 2023-11-24T06:08:26.901 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.896+0000 7fed9cff9700 1 --2- 172.21.15.91:0/2423287118 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fed8006aed0 0x7fed8006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:26.901 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.896+0000 7fed9cff9700 1 -- 172.21.15.91:0/2423287118 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed980895e0 con 0x7feda01478c0 2023-11-24T06:08:26.901 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.896+0000 7fed9f7fe700 1 --2- 172.21.15.91:0/2423287118 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fed8006aed0 0x7fed8006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:26.901 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.896+0000 7fed9f7fe700 1 --2- 172.21.15.91:0/2423287118 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fed8006aed0 0x7fed8006d390 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fed90006b20 tx=0x7fed90008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:26.903 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:26.900+0000 7fed9cff9700 1 -- 172.21.15.91:0/2423287118 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed98054690 con 0x7feda01478c0 2023-11-24T06:08:27.091 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.088+0000 7feda5b4b700 1 -- 172.21.15.91:0/2423287118 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feda0075480 con 0x7feda01478c0 2023-11-24T06:08:27.091 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.088+0000 7fed9cff9700 1 -- 172.21.15.91:0/2423287118 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fed98057cb0 con 0x7feda01478c0 2023-11-24T06:08:27.091 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 32 2023-11-24T06:08:27.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.090+0000 7feda5b4b700 1 -- 172.21.15.91:0/2423287118 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fed8006aed0 msgr2=0x7fed8006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:27.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.090+0000 7feda5b4b700 1 --2- 172.21.15.91:0/2423287118 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fed8006aed0 0x7fed8006d390 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fed90006b20 tx=0x7fed90008040 comp rx=0 tx=0).stop 2023-11-24T06:08:27.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.090+0000 7feda5b4b700 1 -- 172.21.15.91:0/2423287118 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feda01478c0 msgr2=0x7feda01e4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:27.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.090+0000 7feda5b4b700 1 --2- 172.21.15.91:0/2423287118 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feda01478c0 0x7feda01e4b70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fed9800cb50 tx=0x7fed9800ce60 comp rx=0 tx=0).stop 2023-11-24T06:08:27.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.090+0000 7feda5b4b700 1 -- 172.21.15.91:0/2423287118 shutdown_connections 2023-11-24T06:08:27.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.090+0000 7feda5b4b700 1 --2- 172.21.15.91:0/2423287118 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fed8006aed0 0x7fed8006d390 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.090+0000 7feda5b4b700 1 --2- 172.21.15.91:0/2423287118 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7feda006b620 0x7feda01e1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.090+0000 7feda5b4b700 1 --2- 172.21.15.91:0/2423287118 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7feda01478c0 0x7feda01e4b70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.090+0000 7feda5b4b700 1 --2- 172.21.15.91:0/2423287118 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7feda00b5ab0 0x7feda01e4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.090+0000 7feda5b4b700 1 -- 172.21.15.91:0/2423287118 >> 172.21.15.91:0/2423287118 conn(0x7feda00b2740 msgr2=0x7feda01dbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:27.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.090+0000 7feda5b4b700 1 -- 172.21.15.91:0/2423287118 shutdown_connections 2023-11-24T06:08:27.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.090+0000 7feda5b4b700 1 -- 172.21.15.91:0/2423287118 wait complete. 2023-11-24T06:08:27.105 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:08:27.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.227+0000 7fb24e6e0700 1 Processor -- start 2023-11-24T06:08:27.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.227+0000 7fb24e6e0700 1 -- start start 2023-11-24T06:08:27.232 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.228+0000 7fb24e6e0700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fb24815a440 0x7fb2480cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.232 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.228+0000 7fb24e6e0700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb24815a820 0x7fb2480d04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.232 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.228+0000 7fb24e6e0700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2480b05f0 con 0x7fb24815eb80 2023-11-24T06:08:27.232 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.228+0000 7fb24e6e0700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2480673f0 con 0x7fb24815a440 2023-11-24T06:08:27.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.228+0000 7fb24e6e0700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb24815f080 con 0x7fb24815a820 2023-11-24T06:08:27.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.228+0000 7fb247fff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fb24815a440 0x7fb2480cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.228+0000 7fb247fff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fb24815a440 0x7fb2480cffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58640/0 (socket says 172.21.15.91:58640) 2023-11-24T06:08:27.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.228+0000 7fb247fff700 1 -- 172.21.15.91:0/3351400111 learned_addr learned my addr 172.21.15.91:0/3351400111 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:27.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.228+0000 7fb2477fe700 1 --2- 172.21.15.91:0/3351400111 >> v2:172.21.15.94:3300/0 conn(0x7fb24815a820 0x7fb2480d04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.228+0000 7fb247fff700 1 -- 172.21.15.91:0/3351400111 >> v2:172.21.15.94:3300/0 conn(0x7fb24815a820 msgr2=0x7fb2480d04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:27.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.228+0000 7fb247fff700 1 --2- 172.21.15.91:0/3351400111 >> v2:172.21.15.94:3300/0 conn(0x7fb24815a820 0x7fb2480d04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.228+0000 7fb247fff700 1 -- 172.21.15.91:0/3351400111 >> v1:172.21.15.94:6789/0 conn(0x7fb24815eb80 legacy=0x7fb248066ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:27.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.228+0000 7fb247fff700 1 -- 172.21.15.91:0/3351400111 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2480afd50 con 0x7fb24815a440 2023-11-24T06:08:27.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.228+0000 7fb247fff700 1 --2- 172.21.15.91:0/3351400111 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb24815a440 0x7fb2480cffa0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb248069460 tx=0x7fb23c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac292832537a33e2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:27.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.228+0000 7fb246ffd700 1 -- 172.21.15.91:0/3351400111 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c004230 con 0x7fb24815a440 2023-11-24T06:08:27.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.228+0000 7fb246ffd700 1 -- 172.21.15.91:0/3351400111 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23c004390 con 0x7fb24815a440 2023-11-24T06:08:27.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.229+0000 7fb246ffd700 1 -- 172.21.15.91:0/3351400111 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c0036a0 con 0x7fb24815a440 2023-11-24T06:08:27.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.229+0000 7fb24e6e0700 1 -- 172.21.15.91:0/3351400111 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb24815a440 msgr2=0x7fb2480cffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:27.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.229+0000 7fb24e6e0700 1 --2- 172.21.15.91:0/3351400111 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb24815a440 0x7fb2480cffa0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb248069460 tx=0x7fb23c009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.229+0000 7fb24e6e0700 1 -- 172.21.15.91:0/3351400111 shutdown_connections 2023-11-24T06:08:27.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.229+0000 7fb24e6e0700 1 --2- 172.21.15.91:0/3351400111 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb24815a440 0x7fb2480cffa0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.229+0000 7fb24e6e0700 1 --2- 172.21.15.91:0/3351400111 >> v2:172.21.15.94:3300/0 conn(0x7fb24815a820 0x7fb2480d04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.229+0000 7fb24e6e0700 1 -- 172.21.15.91:0/3351400111 >> 172.21.15.91:0/3351400111 conn(0x7fb2480b5070 msgr2=0x7fb2480b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:27.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.229+0000 7fb24e6e0700 1 -- 172.21.15.91:0/3351400111 shutdown_connections 2023-11-24T06:08:27.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.229+0000 7fb24e6e0700 1 -- 172.21.15.91:0/3351400111 wait complete. 2023-11-24T06:08:27.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.230+0000 7fb24e6e0700 1 Processor -- start 2023-11-24T06:08:27.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.230+0000 7fb24e6e0700 1 -- start start 2023-11-24T06:08:27.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.231+0000 7fb24e6e0700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb24815a440 0x7fb2481e3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.231+0000 7fb24e6e0700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb24815eb80 0x7fb248142500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.231+0000 7fb24e6e0700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb248150560 0x7fb2481e5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.231+0000 7fb24e6e0700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2481e47e0 con 0x7fb24815eb80 2023-11-24T06:08:27.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.231+0000 7fb24e6e0700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2481e4ac0 con 0x7fb248150560 2023-11-24T06:08:27.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.231+0000 7fb24e6e0700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2481e4da0 con 0x7fb24815a440 2023-11-24T06:08:27.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.231+0000 7fb2477fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb24815eb80 0x7fb248142500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.231+0000 7fb2477fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb24815eb80 0x7fb248142500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58648/0 (socket says 172.21.15.91:58648) 2023-11-24T06:08:27.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.231+0000 7fb2477fe700 1 -- 172.21.15.91:0/210776842 learned_addr learned my addr 172.21.15.91:0/210776842 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:27.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.231+0000 7fb247fff700 1 --2- 172.21.15.91:0/210776842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb24815a440 0x7fb2481e3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.231+0000 7fb24cc7d700 1 --2- 172.21.15.91:0/210776842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb248150560 0x7fb2481e5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.231+0000 7fb2477fe700 1 -- 172.21.15.91:0/210776842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb24815a440 msgr2=0x7fb2481e3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:27.240 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.231+0000 7fb2477fe700 1 --2- 172.21.15.91:0/210776842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb24815a440 0x7fb2481e3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.240 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.232+0000 7fb2477fe700 1 -- 172.21.15.91:0/210776842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb248150560 msgr2=0x7fb2481e5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:27.240 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.232+0000 7fb2477fe700 1 --2- 172.21.15.91:0/210776842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb248150560 0x7fb2481e5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.240 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.232+0000 7fb2477fe700 1 -- 172.21.15.91:0/210776842 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c009550 con 0x7fb24815eb80 2023-11-24T06:08:27.240 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.232+0000 7fb24cc7d700 1 --2- 172.21.15.91:0/210776842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb248150560 0x7fb2481e5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:27.240 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.232+0000 7fb247fff700 1 --2- 172.21.15.91:0/210776842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb24815a440 0x7fb2481e3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:27.240 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.232+0000 7fb2477fe700 1 --2- 172.21.15.91:0/210776842 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb24815eb80 0x7fb248142500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb23800c7f0 tx=0x7fb23800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:27.240 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.232+0000 7fb244ff9700 1 -- 172.21.15.91:0/210776842 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb238004220 con 0x7fb24815eb80 2023-11-24T06:08:27.240 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.232+0000 7fb244ff9700 1 -- 172.21.15.91:0/210776842 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb238004380 con 0x7fb24815eb80 2023-11-24T06:08:27.240 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.232+0000 7fb24e6e0700 1 -- 172.21.15.91:0/210776842 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2480cce90 con 0x7fb24815eb80 2023-11-24T06:08:27.240 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.232+0000 7fb244ff9700 1 -- 172.21.15.91:0/210776842 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2380036a0 con 0x7fb24815eb80 2023-11-24T06:08:27.240 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.232+0000 7fb24e6e0700 1 -- 172.21.15.91:0/210776842 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb248142a40 con 0x7fb24815eb80 2023-11-24T06:08:27.240 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.233+0000 7fb24e6e0700 1 -- 172.21.15.91:0/210776842 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2480cccf0 con 0x7fb24815eb80 2023-11-24T06:08:27.241 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.233+0000 7fb244ff9700 1 -- 172.21.15.91:0/210776842 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb238003800 con 0x7fb24815eb80 2023-11-24T06:08:27.241 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.234+0000 7fb244ff9700 1 --2- 172.21.15.91:0/210776842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb23406aed0 0x7fb23406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.241 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.234+0000 7fb244ff9700 1 -- 172.21.15.91:0/210776842 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb238089200 con 0x7fb24815eb80 2023-11-24T06:08:27.241 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.234+0000 7fb247fff700 1 --2- 172.21.15.91:0/210776842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb23406aed0 0x7fb23406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.241 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.234+0000 7fb247fff700 1 --2- 172.21.15.91:0/210776842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb23406aed0 0x7fb23406d390 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009130 tx=0x7fb23c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:27.241 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.238+0000 7fb244ff9700 1 -- 172.21.15.91:0/210776842 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2480cccf0 con 0x7fb24815eb80 2023-11-24T06:08:27.431 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.428+0000 7fb24e6e0700 1 -- 172.21.15.91:0/210776842 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb2480cccf0 con 0x7fb24815eb80 2023-11-24T06:08:27.432 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.429+0000 7fb244ff9700 1 -- 172.21.15.91:0/210776842 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb2480cccf0 con 0x7fb24815eb80 2023-11-24T06:08:27.432 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 64 2023-11-24T06:08:27.434 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.431+0000 7fb24e6e0700 1 -- 172.21.15.91:0/210776842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb23406aed0 msgr2=0x7fb23406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:27.434 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.431+0000 7fb24e6e0700 1 --2- 172.21.15.91:0/210776842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb23406aed0 0x7fb23406d390 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009130 tx=0x7fb23c008030 comp rx=0 tx=0).stop 2023-11-24T06:08:27.434 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.431+0000 7fb24e6e0700 1 -- 172.21.15.91:0/210776842 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb24815eb80 msgr2=0x7fb248142500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:27.434 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.431+0000 7fb24e6e0700 1 --2- 172.21.15.91:0/210776842 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb24815eb80 0x7fb248142500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb23800c7f0 tx=0x7fb23800cbb0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.435 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.431+0000 7fb24e6e0700 1 -- 172.21.15.91:0/210776842 shutdown_connections 2023-11-24T06:08:27.435 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.431+0000 7fb24e6e0700 1 --2- 172.21.15.91:0/210776842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb23406aed0 0x7fb23406d390 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.435 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.431+0000 7fb24e6e0700 1 --2- 172.21.15.91:0/210776842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb248150560 0x7fb2481e5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.435 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.431+0000 7fb24e6e0700 1 --2- 172.21.15.91:0/210776842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb24815a440 0x7fb2481e3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.435 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.431+0000 7fb24e6e0700 1 --2- 172.21.15.91:0/210776842 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb24815eb80 0x7fb248142500 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.435 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.431+0000 7fb24e6e0700 1 -- 172.21.15.91:0/210776842 >> 172.21.15.91:0/210776842 conn(0x7fb2480b5070 msgr2=0x7fb2480b2730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:27.435 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.431+0000 7fb24e6e0700 1 -- 172.21.15.91:0/210776842 shutdown_connections 2023-11-24T06:08:27.435 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.431+0000 7fb24e6e0700 1 -- 172.21.15.91:0/210776842 wait complete. 2023-11-24T06:08:27.446 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:27.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.563+0000 7f0e70b74700 1 Processor -- start 2023-11-24T06:08:27.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.563+0000 7f0e70b74700 1 -- start start 2023-11-24T06:08:27.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e70b74700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f0e6c1458c0 0x7f0e6c070c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e70b74700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0e6c15f940 0x7f0e6c15fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e70b74700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e6c0caa20 0x7f0e6c0caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e70b74700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e6c160310 con 0x7f0e6c1458c0 2023-11-24T06:08:27.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e70b74700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e6c0c9fd0 con 0x7f0e6c15f940 2023-11-24T06:08:27.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e70b74700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c06e960 con 0x7f0e6c0caa20 2023-11-24T06:08:27.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e6a59c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f0e6c1458c0 0x7f0e6c070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e6a59c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f0e6c1458c0 0x7f0e6c070c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58654/0 (socket says 172.21.15.91:58654) 2023-11-24T06:08:27.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e6a59c700 1 -- 172.21.15.91:0/76123421 learned_addr learned my addr 172.21.15.91:0/76123421 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:27.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e69d9b700 1 --2- 172.21.15.91:0/76123421 >> v2:172.21.15.94:3300/0 conn(0x7f0e6c15f940 0x7f0e6c15fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e6ad9d700 1 --2- 172.21.15.91:0/76123421 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e6c0caa20 0x7f0e6c0caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e6a59c700 1 -- 172.21.15.91:0/76123421 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e6c0caa20 msgr2=0x7f0e6c0caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:27.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e6a59c700 1 --2- 172.21.15.91:0/76123421 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e6c0caa20 0x7f0e6c0caeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e6a59c700 1 -- 172.21.15.91:0/76123421 >> v2:172.21.15.94:3300/0 conn(0x7f0e6c15f940 msgr2=0x7f0e6c15fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:27.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e6a59c700 1 --2- 172.21.15.91:0/76123421 >> v2:172.21.15.94:3300/0 conn(0x7f0e6c15f940 0x7f0e6c15fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e6a59c700 1 -- 172.21.15.91:0/76123421 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c06ec40 con 0x7f0e6c1458c0 2023-11-24T06:08:27.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e69d9b700 1 --2- 172.21.15.91:0/76123421 >> v2:172.21.15.94:3300/0 conn(0x7f0e6c15f940 0x7f0e6c15fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:27.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.564+0000 7f0e6a59c700 1 --2- 172.21.15.91:0/76123421 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e6c1458c0 0x7f0e6c070c80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c15f8a0 tx=0x7f0e5c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b9b9f179d6b059 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:27.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.565+0000 7f0e6959a700 1 -- 172.21.15.91:0/76123421 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c00b870 con 0x7f0e6c1458c0 2023-11-24T06:08:27.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.565+0000 7f0e6959a700 1 -- 172.21.15.91:0/76123421 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c00b9d0 con 0x7f0e6c1458c0 2023-11-24T06:08:27.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.565+0000 7f0e6959a700 1 -- 172.21.15.91:0/76123421 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c00bd60 con 0x7f0e6c1458c0 2023-11-24T06:08:27.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.565+0000 7f0e70b74700 1 -- 172.21.15.91:0/76123421 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e6c1458c0 msgr2=0x7f0e6c070c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:27.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.565+0000 7f0e70b74700 1 --2- 172.21.15.91:0/76123421 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e6c1458c0 0x7f0e6c070c80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c15f8a0 tx=0x7f0e5c009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.565+0000 7f0e70b74700 1 -- 172.21.15.91:0/76123421 shutdown_connections 2023-11-24T06:08:27.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.565+0000 7f0e70b74700 1 --2- 172.21.15.91:0/76123421 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e6c0caa20 0x7f0e6c0caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.565+0000 7f0e70b74700 1 --2- 172.21.15.91:0/76123421 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e6c1458c0 0x7f0e6c070c80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.565+0000 7f0e70b74700 1 --2- 172.21.15.91:0/76123421 >> v2:172.21.15.94:3300/0 conn(0x7f0e6c15f940 0x7f0e6c15fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.565+0000 7f0e70b74700 1 -- 172.21.15.91:0/76123421 >> 172.21.15.91:0/76123421 conn(0x7f0e6c0cf980 msgr2=0x7f0e6c0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:27.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.565+0000 7f0e70b74700 1 -- 172.21.15.91:0/76123421 shutdown_connections 2023-11-24T06:08:27.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.565+0000 7f0e70b74700 1 -- 172.21.15.91:0/76123421 wait complete. 2023-11-24T06:08:27.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.567+0000 7f0e70b74700 1 Processor -- start 2023-11-24T06:08:27.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.567+0000 7f0e70b74700 1 -- start start 2023-11-24T06:08:27.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.567+0000 7f0e70b74700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e6c0caa20 0x7f0e6c15c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.567+0000 7f0e70b74700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e6c1458c0 0x7f0e6c13a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.567+0000 7f0e70b74700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e6c15f940 0x7f0e6c15b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.567+0000 7f0e70b74700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c15d670 con 0x7f0e6c1458c0 2023-11-24T06:08:27.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.567+0000 7f0e70b74700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c13af00 con 0x7f0e6c0caa20 2023-11-24T06:08:27.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.567+0000 7f0e70b74700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c15b5a0 con 0x7f0e6c15f940 2023-11-24T06:08:27.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.567+0000 7f0e69d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e6c1458c0 0x7f0e6c13a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.567+0000 7f0e69d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e6c1458c0 0x7f0e6c13a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58662/0 (socket says 172.21.15.91:58662) 2023-11-24T06:08:27.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.567+0000 7f0e69d9b700 1 -- 172.21.15.91:0/490536745 learned_addr learned my addr 172.21.15.91:0/490536745 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:27.575 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.568+0000 7f0e6a59c700 1 --2- 172.21.15.91:0/490536745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e6c0caa20 0x7f0e6c15c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.575 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.568+0000 7f0e6ad9d700 1 --2- 172.21.15.91:0/490536745 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e6c15f940 0x7f0e6c15b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.575 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.568+0000 7f0e69d9b700 1 -- 172.21.15.91:0/490536745 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e6c15f940 msgr2=0x7f0e6c15b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:27.575 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.568+0000 7f0e69d9b700 1 --2- 172.21.15.91:0/490536745 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e6c15f940 0x7f0e6c15b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.575 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.568+0000 7f0e69d9b700 1 -- 172.21.15.91:0/490536745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e6c0caa20 msgr2=0x7f0e6c15c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:27.575 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.568+0000 7f0e69d9b700 1 --2- 172.21.15.91:0/490536745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e6c0caa20 0x7f0e6c15c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.575 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.568+0000 7f0e69d9b700 1 -- 172.21.15.91:0/490536745 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c009550 con 0x7f0e6c1458c0 2023-11-24T06:08:27.575 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.568+0000 7f0e69d9b700 1 --2- 172.21.15.91:0/490536745 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e6c1458c0 0x7f0e6c13a9c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0e6000c820 tx=0x7f0e6000cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:27.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.568+0000 7f0e5affd700 1 -- 172.21.15.91:0/490536745 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e60011800 con 0x7f0e6c1458c0 2023-11-24T06:08:27.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.568+0000 7f0e5affd700 1 -- 172.21.15.91:0/490536745 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e60011960 con 0x7f0e6c1458c0 2023-11-24T06:08:27.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.568+0000 7f0e5affd700 1 -- 172.21.15.91:0/490536745 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e60011cf0 con 0x7f0e6c1458c0 2023-11-24T06:08:27.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.568+0000 7f0e70b74700 1 -- 172.21.15.91:0/490536745 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e6c15ca60 con 0x7f0e6c1458c0 2023-11-24T06:08:27.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.568+0000 7f0e70b74700 1 -- 172.21.15.91:0/490536745 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e6c15d020 con 0x7f0e6c1458c0 2023-11-24T06:08:27.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.569+0000 7f0e70b74700 1 -- 172.21.15.91:0/490536745 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e6c067f60 con 0x7f0e6c1458c0 2023-11-24T06:08:27.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.570+0000 7f0e5affd700 1 -- 172.21.15.91:0/490536745 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0e600233a0 con 0x7f0e6c1458c0 2023-11-24T06:08:27.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.570+0000 7f0e5affd700 1 --2- 172.21.15.91:0/490536745 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0e4c06aed0 0x7f0e4c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.570+0000 7f0e5affd700 1 -- 172.21.15.91:0/490536745 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e60089700 con 0x7f0e6c1458c0 2023-11-24T06:08:27.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.570+0000 7f0e6a59c700 1 --2- 172.21.15.91:0/490536745 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0e4c06aed0 0x7f0e4c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.570+0000 7f0e6a59c700 1 --2- 172.21.15.91:0/490536745 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0e4c06aed0 0x7f0e4c06d390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c000c00 tx=0x7f0e5c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:27.577 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.574+0000 7f0e5affd700 1 -- 172.21.15.91:0/490536745 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e60054840 con 0x7f0e6c1458c0 2023-11-24T06:08:27.770 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.767+0000 7f0e70b74700 1 -- 172.21.15.91:0/490536745 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e6c0b1370 con 0x7f0e6c1458c0 2023-11-24T06:08:27.771 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.768+0000 7f0e5affd700 1 -- 172.21.15.91:0/490536745 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7f0e60057e60 con 0x7f0e6c1458c0 2023-11-24T06:08:27.771 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:27.771 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":20,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:02.415612+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:27.773 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.770+0000 7f0e70b74700 1 -- 172.21.15.91:0/490536745 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0e4c06aed0 msgr2=0x7f0e4c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:27.773 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.770+0000 7f0e70b74700 1 --2- 172.21.15.91:0/490536745 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0e4c06aed0 0x7f0e4c06d390 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c000c00 tx=0x7f0e5c008030 comp rx=0 tx=0).stop 2023-11-24T06:08:27.774 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.770+0000 7f0e70b74700 1 -- 172.21.15.91:0/490536745 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e6c1458c0 msgr2=0x7f0e6c13a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:27.774 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.770+0000 7f0e70b74700 1 --2- 172.21.15.91:0/490536745 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e6c1458c0 0x7f0e6c13a9c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0e6000c820 tx=0x7f0e6000cb30 comp rx=0 tx=0).stop 2023-11-24T06:08:27.774 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.770+0000 7f0e70b74700 1 -- 172.21.15.91:0/490536745 shutdown_connections 2023-11-24T06:08:27.774 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.770+0000 7f0e70b74700 1 --2- 172.21.15.91:0/490536745 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0e4c06aed0 0x7f0e4c06d390 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.774 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.770+0000 7f0e70b74700 1 --2- 172.21.15.91:0/490536745 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e6c0caa20 0x7f0e6c15c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.774 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.770+0000 7f0e70b74700 1 --2- 172.21.15.91:0/490536745 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e6c15f940 0x7f0e6c15b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.774 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.770+0000 7f0e70b74700 1 --2- 172.21.15.91:0/490536745 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e6c1458c0 0x7f0e6c13a9c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.774 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.770+0000 7f0e70b74700 1 -- 172.21.15.91:0/490536745 >> 172.21.15.91:0/490536745 conn(0x7f0e6c0cf980 msgr2=0x7f0e6c0b0300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:27.774 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.770+0000 7f0e70b74700 1 -- 172.21.15.91:0/490536745 shutdown_connections 2023-11-24T06:08:27.774 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.771+0000 7f0e70b74700 1 -- 172.21.15.91:0/490536745 wait complete. 2023-11-24T06:08:27.785 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:07:58.710106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:07:59.774819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:08:27.786 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:08:27.908 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.906+0000 7ff16956b700 1 Processor -- start 2023-11-24T06:08:27.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.906+0000 7ff16956b700 1 -- start start 2023-11-24T06:08:27.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.906+0000 7ff16956b700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff16415d7f0 0x7ff16406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.906+0000 7ff16956b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff16406d800 0x7ff164150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.907+0000 7ff16956b700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1640b39b0 con 0x7ff164151110 2023-11-24T06:08:27.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.907+0000 7ff16956b700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff16406dcb0 con 0x7ff16415d7f0 2023-11-24T06:08:27.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.907+0000 7ff16956b700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1640cf540 con 0x7ff16406d800 2023-11-24T06:08:27.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.907+0000 7ff162ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff16415d7f0 0x7ff16406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.907+0000 7ff1637fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7ff164151110 0x7ff16415df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53606/0 (socket says 172.21.15.91:53606) 2023-11-24T06:08:27.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.907+0000 7ff1637fe700 1 -- 172.21.15.91:0/3687088801 learned_addr learned my addr 172.21.15.91:0/3687088801 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:27.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.907+0000 7ff162ffd700 1 -- 172.21.15.91:0/3687088801 >> v2:172.21.15.94:3300/0 conn(0x7ff16406d800 msgr2=0x7ff164150bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:27.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.907+0000 7ff162ffd700 1 --2- 172.21.15.91:0/3687088801 >> v2:172.21.15.94:3300/0 conn(0x7ff16406d800 0x7ff164150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.907+0000 7ff162ffd700 1 -- 172.21.15.91:0/3687088801 >> v1:172.21.15.91:6789/0 conn(0x7ff164151110 legacy=0x7ff16415df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:27.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.907+0000 7ff161ffb700 1 -- 172.21.15.91:0/3687088801 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990401021 0 0) 0x7ff1640b39b0 con 0x7ff164151110 2023-11-24T06:08:27.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.907+0000 7ff162ffd700 1 -- 172.21.15.91:0/3687088801 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1640b2da0 con 0x7ff16415d7f0 2023-11-24T06:08:27.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.907+0000 7ff162ffd700 1 --2- 172.21.15.91:0/3687088801 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff16415d7f0 0x7ff16406d2c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff16406c2b0 tx=0x7ff154009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50fdf4745d81a0da server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:27.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.907+0000 7ff161ffb700 1 -- 172.21.15.91:0/3687088801 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff154011830 con 0x7ff16415d7f0 2023-11-24T06:08:27.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.907+0000 7ff161ffb700 1 -- 172.21.15.91:0/3687088801 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff154011990 con 0x7ff16415d7f0 2023-11-24T06:08:27.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.907+0000 7ff161ffb700 1 -- 172.21.15.91:0/3687088801 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff154011d20 con 0x7ff16415d7f0 2023-11-24T06:08:27.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.908+0000 7ff16956b700 1 -- 172.21.15.91:0/3687088801 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff16415d7f0 msgr2=0x7ff16406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:27.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.908+0000 7ff16956b700 1 --2- 172.21.15.91:0/3687088801 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff16415d7f0 0x7ff16406d2c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff16406c2b0 tx=0x7ff154009b20 comp rx=0 tx=0).stop 2023-11-24T06:08:27.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.908+0000 7ff16956b700 1 -- 172.21.15.91:0/3687088801 shutdown_connections 2023-11-24T06:08:27.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.908+0000 7ff16956b700 1 --2- 172.21.15.91:0/3687088801 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff16415d7f0 0x7ff16406d2c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.908+0000 7ff16956b700 1 --2- 172.21.15.91:0/3687088801 >> v2:172.21.15.94:3300/0 conn(0x7ff16406d800 0x7ff164150bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.908+0000 7ff16956b700 1 -- 172.21.15.91:0/3687088801 >> 172.21.15.91:0/3687088801 conn(0x7ff1640b1650 msgr2=0x7ff1640b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:27.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.908+0000 7ff16956b700 1 -- 172.21.15.91:0/3687088801 shutdown_connections 2023-11-24T06:08:27.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.908+0000 7ff16956b700 1 -- 172.21.15.91:0/3687088801 wait complete. 2023-11-24T06:08:27.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.909+0000 7ff16956b700 1 Processor -- start 2023-11-24T06:08:27.913 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.910+0000 7ff16956b700 1 -- start start 2023-11-24T06:08:27.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.910+0000 7ff16956b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff16406d800 0x7ff164152f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.910+0000 7ff16956b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff164151110 0x7ff1641501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.910+0000 7ff16956b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff16415d7f0 0x7ff1641e4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.910+0000 7ff16956b700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff164157440 con 0x7ff16415d7f0 2023-11-24T06:08:27.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.910+0000 7ff16956b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff164150730 con 0x7ff16406d800 2023-11-24T06:08:27.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff16956b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff164153530 con 0x7ff164151110 2023-11-24T06:08:27.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff1637fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff16415d7f0 0x7ff1641e4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff1637fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff16415d7f0 0x7ff1641e4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58678/0 (socket says 172.21.15.91:58678) 2023-11-24T06:08:27.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff1637fe700 1 -- 172.21.15.91:0/755095184 learned_addr learned my addr 172.21.15.91:0/755095184 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:27.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff162ffd700 1 --2- 172.21.15.91:0/755095184 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff16406d800 0x7ff164152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff1627fc700 1 --2- 172.21.15.91:0/755095184 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff164151110 0x7ff1641501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff1637fe700 1 -- 172.21.15.91:0/755095184 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff164151110 msgr2=0x7ff1641501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:27.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff1637fe700 1 --2- 172.21.15.91:0/755095184 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff164151110 0x7ff1641501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff1637fe700 1 -- 172.21.15.91:0/755095184 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff16406d800 msgr2=0x7ff164152f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:27.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff1637fe700 1 --2- 172.21.15.91:0/755095184 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff16406d800 0x7ff164152f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:27.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff1637fe700 1 -- 172.21.15.91:0/755095184 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1540094a0 con 0x7ff16415d7f0 2023-11-24T06:08:27.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff162ffd700 1 --2- 172.21.15.91:0/755095184 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff16406d800 0x7ff164152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:27.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff1637fe700 1 --2- 172.21.15.91:0/755095184 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff16415d7f0 0x7ff1641e4940 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff15c002210 tx=0x7ff15c00c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:27.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff14b7fe700 1 -- 172.21.15.91:0/755095184 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c0070d0 con 0x7ff16415d7f0 2023-11-24T06:08:27.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff14b7fe700 1 -- 172.21.15.91:0/755095184 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff15c007230 con 0x7ff16415d7f0 2023-11-24T06:08:27.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff16956b700 1 -- 172.21.15.91:0/755095184 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff164153810 con 0x7ff16415d7f0 2023-11-24T06:08:27.918 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff14b7fe700 1 -- 172.21.15.91:0/755095184 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c0075c0 con 0x7ff16415d7f0 2023-11-24T06:08:27.918 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.911+0000 7ff16956b700 1 -- 172.21.15.91:0/755095184 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1641e4e80 con 0x7ff16415d7f0 2023-11-24T06:08:27.918 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.912+0000 7ff16956b700 1 -- 172.21.15.91:0/755095184 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff164157280 con 0x7ff16415d7f0 2023-11-24T06:08:27.918 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.913+0000 7ff14b7fe700 1 -- 172.21.15.91:0/755095184 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff15c007720 con 0x7ff16415d7f0 2023-11-24T06:08:27.918 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.913+0000 7ff14b7fe700 1 --2- 172.21.15.91:0/755095184 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff15006aed0 0x7ff15006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:27.918 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.913+0000 7ff14b7fe700 1 -- 172.21.15.91:0/755095184 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff15c08c700 con 0x7ff16415d7f0 2023-11-24T06:08:27.918 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.913+0000 7ff162ffd700 1 --2- 172.21.15.91:0/755095184 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff15006aed0 0x7ff15006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:27.918 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.913+0000 7ff162ffd700 1 --2- 172.21.15.91:0/755095184 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff15006aed0 0x7ff15006d390 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff1540090b0 tx=0x7ff154008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:27.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:27.917+0000 7ff14b7fe700 1 -- 172.21.15.91:0/755095184 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff164157280 con 0x7ff16415d7f0 2023-11-24T06:08:28.114 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.111+0000 7ff16956b700 1 -- 172.21.15.91:0/755095184 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff164157280 con 0x7ff16415d7f0 2023-11-24T06:08:28.114 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.111+0000 7ff14b7fe700 1 -- 172.21.15.91:0/755095184 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff164157280 con 0x7ff16415d7f0 2023-11-24T06:08:28.114 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:08:28.116 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.113+0000 7ff16956b700 1 -- 172.21.15.91:0/755095184 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff15006aed0 msgr2=0x7ff15006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.113+0000 7ff16956b700 1 --2- 172.21.15.91:0/755095184 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff15006aed0 0x7ff15006d390 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff1540090b0 tx=0x7ff154008030 comp rx=0 tx=0).stop 2023-11-24T06:08:28.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.113+0000 7ff16956b700 1 -- 172.21.15.91:0/755095184 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff16415d7f0 msgr2=0x7ff1641e4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.113+0000 7ff16956b700 1 --2- 172.21.15.91:0/755095184 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff16415d7f0 0x7ff1641e4940 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff15c002210 tx=0x7ff15c00c550 comp rx=0 tx=0).stop 2023-11-24T06:08:28.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.113+0000 7ff16956b700 1 -- 172.21.15.91:0/755095184 shutdown_connections 2023-11-24T06:08:28.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.113+0000 7ff16956b700 1 --2- 172.21.15.91:0/755095184 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff15006aed0 0x7ff15006d390 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.113+0000 7ff16956b700 1 --2- 172.21.15.91:0/755095184 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff16406d800 0x7ff164152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.113+0000 7ff16956b700 1 --2- 172.21.15.91:0/755095184 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff16415d7f0 0x7ff1641e4940 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.113+0000 7ff16956b700 1 --2- 172.21.15.91:0/755095184 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff164151110 0x7ff1641501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.113+0000 7ff16956b700 1 -- 172.21.15.91:0/755095184 >> 172.21.15.91:0/755095184 conn(0x7ff1640b1650 msgr2=0x7ff16415c730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:28.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.114+0000 7ff16956b700 1 -- 172.21.15.91:0/755095184 shutdown_connections 2023-11-24T06:08:28.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.114+0000 7ff16956b700 1 -- 172.21.15.91:0/755095184 wait complete. 2023-11-24T06:08:28.129 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:08:28.253 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.250+0000 7f30a28b3700 1 Processor -- start 2023-11-24T06:08:28.253 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.250+0000 7f30a28b3700 1 -- start start 2023-11-24T06:08:28.255 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f30a28b3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f309c15ad50 0x7f309c0b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.255 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f30a28b3700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f309c0b3790 0x7f309c0b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.255 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f30a28b3700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f309c15b130 con 0x7f309c0cc090 2023-11-24T06:08:28.255 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f30a28b3700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f309c072ce0 con 0x7f309c0b3790 2023-11-24T06:08:28.255 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f30a28b3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0b4160 con 0x7f309c15ad50 2023-11-24T06:08:28.255 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f30a0e50700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f309c0b3790 0x7f309c0b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.255 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f30a0e50700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f309c0b3790 0x7f309c0b3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58684/0 (socket says 172.21.15.91:58684) 2023-11-24T06:08:28.255 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f30a0e50700 1 -- 172.21.15.91:0/1628255036 learned_addr learned my addr 172.21.15.91:0/1628255036 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:28.256 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f309b7fe700 1 --1- 172.21.15.91:0/1628255036 >> v1:172.21.15.91:6789/0 conn(0x7f309c0cc090 0x7f309c0725d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53616/0 (socket says 172.21.15.91:53616) 2023-11-24T06:08:28.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f309bfff700 1 --2- 172.21.15.91:0/1628255036 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f309c15ad50 0x7f309c0b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f30a0e50700 1 -- 172.21.15.91:0/1628255036 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f309c15ad50 msgr2=0x7f309c0b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:28.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f30a0e50700 1 --2- 172.21.15.91:0/1628255036 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f309c15ad50 0x7f309c0b2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f30a0e50700 1 -- 172.21.15.91:0/1628255036 >> v1:172.21.15.91:6789/0 conn(0x7f309c0cc090 legacy=0x7f309c0725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f30a0e50700 1 -- 172.21.15.91:0/1628255036 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f309c0cc510 con 0x7f309c0b3790 2023-11-24T06:08:28.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f309affd700 1 -- 172.21.15.91:0/1628255036 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142991984 0 0) 0x7f309c15b130 con 0x7f309c0cc090 2023-11-24T06:08:28.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f30a0e50700 1 --2- 172.21.15.91:0/1628255036 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f309c0b3790 0x7f309c0b3c20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3094009e80 tx=0x7f309400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22872eb1a7f8fa86 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:28.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f309affd700 1 -- 172.21.15.91:0/1628255036 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309400e070 con 0x7f309c0b3790 2023-11-24T06:08:28.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f309affd700 1 -- 172.21.15.91:0/1628255036 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309400b980 con 0x7f309c0b3790 2023-11-24T06:08:28.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.251+0000 7f309affd700 1 -- 172.21.15.91:0/1628255036 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309400bd30 con 0x7f309c0b3790 2023-11-24T06:08:28.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.252+0000 7f30a28b3700 1 -- 172.21.15.91:0/1628255036 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f309c0b3790 msgr2=0x7f309c0b3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.252+0000 7f30a28b3700 1 --2- 172.21.15.91:0/1628255036 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f309c0b3790 0x7f309c0b3c20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3094009e80 tx=0x7f309400b330 comp rx=0 tx=0).stop 2023-11-24T06:08:28.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.252+0000 7f30a28b3700 1 -- 172.21.15.91:0/1628255036 shutdown_connections 2023-11-24T06:08:28.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.252+0000 7f30a28b3700 1 --2- 172.21.15.91:0/1628255036 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f309c0b3790 0x7f309c0b3c20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.252+0000 7f30a28b3700 1 --2- 172.21.15.91:0/1628255036 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f309c15ad50 0x7f309c0b2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.252+0000 7f30a28b3700 1 -- 172.21.15.91:0/1628255036 >> 172.21.15.91:0/1628255036 conn(0x7f309c0b1650 msgr2=0x7f309c0b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:28.258 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.252+0000 7f30a28b3700 1 -- 172.21.15.91:0/1628255036 shutdown_connections 2023-11-24T06:08:28.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.252+0000 7f30a28b3700 1 -- 172.21.15.91:0/1628255036 wait complete. 2023-11-24T06:08:28.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.253+0000 7f30a28b3700 1 Processor -- start 2023-11-24T06:08:28.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.253+0000 7f30a28b3700 1 -- start start 2023-11-24T06:08:28.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f30a28b3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f309c0b3790 0x7f309c151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f30a28b3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f309c0cc090 0x7f309c15def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f30a28b3700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f309c15ad50 0x7f309c150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f30a28b3700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c0ceb30 con 0x7f309c15ad50 2023-11-24T06:08:28.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f30a28b3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309c150e90 con 0x7f309c0cc090 2023-11-24T06:08:28.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f30a28b3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f309c15e560 con 0x7f309c0b3790 2023-11-24T06:08:28.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f30a0e50700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f309c15ad50 0x7f309c150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f30a0e50700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f309c15ad50 0x7f309c150950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58690/0 (socket says 172.21.15.91:58690) 2023-11-24T06:08:28.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f30a0e50700 1 -- 172.21.15.91:0/1628620671 learned_addr learned my addr 172.21.15.91:0/1628620671 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:28.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f309b7fe700 1 --2- 172.21.15.91:0/1628620671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f309c0cc090 0x7f309c15def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f309bfff700 1 --2- 172.21.15.91:0/1628620671 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f309c0b3790 0x7f309c151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f30a0e50700 1 -- 172.21.15.91:0/1628620671 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f309c0b3790 msgr2=0x7f309c151070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f30a0e50700 1 --2- 172.21.15.91:0/1628620671 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f309c0b3790 0x7f309c151070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f30a0e50700 1 -- 172.21.15.91:0/1628620671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f309c0cc090 msgr2=0x7f309c15def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f30a0e50700 1 --2- 172.21.15.91:0/1628620671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f309c0cc090 0x7f309c15def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f30a0e50700 1 -- 172.21.15.91:0/1628620671 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3094009b30 con 0x7f309c15ad50 2023-11-24T06:08:28.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.254+0000 7f309b7fe700 1 --2- 172.21.15.91:0/1628620671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f309c0cc090 0x7f309c15def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:28.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.255+0000 7f309bfff700 1 --2- 172.21.15.91:0/1628620671 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f309c0b3790 0x7f309c151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:28.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.255+0000 7f30a0e50700 1 --2- 172.21.15.91:0/1628620671 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f309c15ad50 0x7f309c150950 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f309400b730 tx=0x7f3094004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:28.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.255+0000 7f3098ff9700 1 -- 172.21.15.91:0/1628620671 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309401c640 con 0x7f309c15ad50 2023-11-24T06:08:28.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.255+0000 7f3098ff9700 1 -- 172.21.15.91:0/1628620671 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309401c7c0 con 0x7f309c15ad50 2023-11-24T06:08:28.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.255+0000 7f3098ff9700 1 -- 172.21.15.91:0/1628620671 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f309401cb70 con 0x7f309c15ad50 2023-11-24T06:08:28.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.255+0000 7f30a28b3700 1 -- 172.21.15.91:0/1628620671 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f309c15e840 con 0x7f309c15ad50 2023-11-24T06:08:28.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.255+0000 7f30a28b3700 1 -- 172.21.15.91:0/1628620671 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f309c15ee00 con 0x7f309c15ad50 2023-11-24T06:08:28.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.256+0000 7f30a28b3700 1 -- 172.21.15.91:0/1628620671 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309c0cee10 con 0x7f309c15ad50 2023-11-24T06:08:28.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.256+0000 7f3098ff9700 1 -- 172.21.15.91:0/1628620671 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f309401ccd0 con 0x7f309c15ad50 2023-11-24T06:08:28.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.257+0000 7f3098ff9700 1 --2- 172.21.15.91:0/1628620671 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f308806aed0 0x7f308806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.257+0000 7f3098ff9700 1 -- 172.21.15.91:0/1628620671 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f309408b020 con 0x7f309c15ad50 2023-11-24T06:08:28.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.257+0000 7f309bfff700 1 --2- 172.21.15.91:0/1628620671 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f308806aed0 0x7f308806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.263 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.257+0000 7f309bfff700 1 --2- 172.21.15.91:0/1628620671 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f308806aed0 0x7f308806d390 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f308c006fd0 tx=0x7f308c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:28.264 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.261+0000 7f3098ff9700 1 -- 172.21.15.91:0/1628620671 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3094056160 con 0x7f309c15ad50 2023-11-24T06:08:28.453 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.451+0000 7f30a28b3700 1 -- 172.21.15.91:0/1628620671 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f309c075480 con 0x7f309c15ad50 2023-11-24T06:08:28.454 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.451+0000 7f3098ff9700 1 -- 172.21.15.91:0/1628620671 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f309402c020 con 0x7f309c15ad50 2023-11-24T06:08:28.454 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 32 2023-11-24T06:08:28.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.453+0000 7f30a28b3700 1 -- 172.21.15.91:0/1628620671 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f308806aed0 msgr2=0x7f308806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.453+0000 7f30a28b3700 1 --2- 172.21.15.91:0/1628620671 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f308806aed0 0x7f308806d390 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f308c006fd0 tx=0x7f308c008040 comp rx=0 tx=0).stop 2023-11-24T06:08:28.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.453+0000 7f30a28b3700 1 -- 172.21.15.91:0/1628620671 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f309c15ad50 msgr2=0x7f309c150950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.453+0000 7f30a28b3700 1 --2- 172.21.15.91:0/1628620671 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f309c15ad50 0x7f309c150950 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f309400b730 tx=0x7f3094004ad0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.453+0000 7f30a28b3700 1 -- 172.21.15.91:0/1628620671 shutdown_connections 2023-11-24T06:08:28.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.453+0000 7f30a28b3700 1 --2- 172.21.15.91:0/1628620671 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f308806aed0 0x7f308806d390 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.453+0000 7f30a28b3700 1 --2- 172.21.15.91:0/1628620671 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f309c15ad50 0x7f309c150950 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.453+0000 7f30a28b3700 1 --2- 172.21.15.91:0/1628620671 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f309c0b3790 0x7f309c151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.453+0000 7f30a28b3700 1 --2- 172.21.15.91:0/1628620671 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f309c0cc090 0x7f309c15def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.453+0000 7f30a28b3700 1 -- 172.21.15.91:0/1628620671 >> 172.21.15.91:0/1628620671 conn(0x7f309c0b1650 msgr2=0x7f309c0725d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:28.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.453+0000 7f30a28b3700 1 -- 172.21.15.91:0/1628620671 shutdown_connections 2023-11-24T06:08:28.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.453+0000 7f30a28b3700 1 -- 172.21.15.91:0/1628620671 wait complete. 2023-11-24T06:08:28.468 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:08:28.591 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.588+0000 7f15c68cf700 1 Processor -- start 2023-11-24T06:08:28.591 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.588+0000 7f15c68cf700 1 -- start start 2023-11-24T06:08:28.594 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.589+0000 7f15c68cf700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15c006b620 0x7f15c00b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.589+0000 7f15c68cf700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f15c006ee60 0x7f15c00b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.589+0000 7f15c68cf700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c0071f50 con 0x7f15c00b5ab0 2023-11-24T06:08:28.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.589+0000 7f15c68cf700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15c0072230 con 0x7f15c006ee60 2023-11-24T06:08:28.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.589+0000 7f15c68cf700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c0072510 con 0x7f15c006b620 2023-11-24T06:08:28.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.589+0000 7f15bf7fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f15c00b5ab0 0x7f15c0161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53628/0 (socket says 172.21.15.91:53628) 2023-11-24T06:08:28.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.589+0000 7f15bf7fe700 1 -- 172.21.15.91:0/440890451 learned_addr learned my addr 172.21.15.91:0/440890451 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:28.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15bffff700 1 --2- 172.21.15.91:0/440890451 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15c006b620 0x7f15c00b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15c4e6c700 1 --2- 172.21.15.91:0/440890451 >> v2:172.21.15.94:3300/0 conn(0x7f15c006ee60 0x7f15c00b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15beffd700 1 -- 172.21.15.91:0/440890451 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028946800 0 0) 0x7f15c0071f50 con 0x7f15c00b5ab0 2023-11-24T06:08:28.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15beffd700 1 -- 172.21.15.91:0/440890451 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15ac0035b0 con 0x7f15c00b5ab0 2023-11-24T06:08:28.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15beffd700 1 -- 172.21.15.91:0/440890451 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 847181971 0 0) 0x7f15ac0035b0 con 0x7f15c00b5ab0 2023-11-24T06:08:28.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15beffd700 1 -- 172.21.15.91:0/440890451 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15c006b620 msgr2=0x7f15c00b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:28.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15beffd700 1 --2- 172.21.15.91:0/440890451 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15c006b620 0x7f15c00b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15beffd700 1 -- 172.21.15.91:0/440890451 >> v2:172.21.15.94:3300/0 conn(0x7f15c006ee60 msgr2=0x7f15c00b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:28.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15beffd700 1 --2- 172.21.15.91:0/440890451 >> v2:172.21.15.94:3300/0 conn(0x7f15c006ee60 0x7f15c00b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15beffd700 1 -- 172.21.15.91:0/440890451 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c00727f0 con 0x7f15c00b5ab0 2023-11-24T06:08:28.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15beffd700 1 -- 172.21.15.91:0/440890451 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f15b0001670 con 0x7f15c00b5ab0 2023-11-24T06:08:28.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15beffd700 1 -- 172.21.15.91:0/440890451 >> v1:172.21.15.91:6789/0 conn(0x7f15c00b5ab0 legacy=0x7f15c0161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15beffd700 1 --2- 172.21.15.91:0/440890451 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15ac003a00 0x7f15ac003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15beffd700 1 --2- 172.21.15.91:0/440890451 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15c006b620 0x7f15ac004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15beffd700 1 --2- 172.21.15.91:0/440890451 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15ac004d90 0x7f15ac005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15beffd700 1 -- 172.21.15.91:0/440890451 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c0072230 con 0x7f15ac004d90 2023-11-24T06:08:28.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15beffd700 1 -- 172.21.15.91:0/440890451 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c0072510 con 0x7f15ac003a00 2023-11-24T06:08:28.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15beffd700 1 -- 172.21.15.91:0/440890451 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ac00ba10 con 0x7f15c006b620 2023-11-24T06:08:28.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15c4e6c700 1 --2- 172.21.15.91:0/440890451 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15ac004d90 0x7f15ac005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15bf7fe700 1 --2- 172.21.15.91:0/440890451 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15c006b620 0x7f15ac004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15bffff700 1 --2- 172.21.15.91:0/440890451 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15ac003a00 0x7f15ac003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15c4e6c700 1 -- 172.21.15.91:0/440890451 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15c006b620 msgr2=0x7f15ac004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:28.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15c4e6c700 1 --2- 172.21.15.91:0/440890451 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15c006b620 0x7f15ac004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15c4e6c700 1 -- 172.21.15.91:0/440890451 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15ac003a00 msgr2=0x7f15ac003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:28.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15c4e6c700 1 --2- 172.21.15.91:0/440890451 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15ac003a00 0x7f15ac003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.590+0000 7f15c4e6c700 1 -- 172.21.15.91:0/440890451 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac0035b0 con 0x7f15ac004d90 2023-11-24T06:08:28.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.591+0000 7f15c4e6c700 1 --2- 172.21.15.91:0/440890451 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15ac004d90 0x7f15ac005210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f15b8009f90 tx=0x7f15ac005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d89b8a2434c4f7d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:28.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.591+0000 7f15beffd700 1 -- 172.21.15.91:0/440890451 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b800ec30 con 0x7f15ac004d90 2023-11-24T06:08:28.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.591+0000 7f15beffd700 1 -- 172.21.15.91:0/440890451 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b800f040 con 0x7f15ac004d90 2023-11-24T06:08:28.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.591+0000 7f15beffd700 1 -- 172.21.15.91:0/440890451 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b8014480 con 0x7f15ac004d90 2023-11-24T06:08:28.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.591+0000 7f15c68cf700 1 -- 172.21.15.91:0/440890451 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15ac004d90 msgr2=0x7f15ac005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.591+0000 7f15c68cf700 1 --2- 172.21.15.91:0/440890451 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15ac004d90 0x7f15ac005210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f15b8009f90 tx=0x7f15ac005880 comp rx=0 tx=0).stop 2023-11-24T06:08:28.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.591+0000 7f15c4e6c700 1 -- 172.21.15.91:0/440890451 reap_dead start 2023-11-24T06:08:28.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.591+0000 7f15c68cf700 1 -- 172.21.15.91:0/440890451 shutdown_connections 2023-11-24T06:08:28.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.591+0000 7f15c68cf700 1 -- 172.21.15.91:0/440890451 >> 172.21.15.91:0/440890451 conn(0x7f15c00b2740 msgr2=0x7f15c00b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:28.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.591+0000 7f15c68cf700 1 -- 172.21.15.91:0/440890451 shutdown_connections 2023-11-24T06:08:28.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.591+0000 7f15c68cf700 1 -- 172.21.15.91:0/440890451 wait complete. 2023-11-24T06:08:28.600 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.593+0000 7f15c68cf700 1 Processor -- start 2023-11-24T06:08:28.600 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.593+0000 7f15c68cf700 1 -- start start 2023-11-24T06:08:28.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.593+0000 7f15c68cf700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c01e2d30 0x7f15c00d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.593+0000 7f15c68cf700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15c00d1010 0x7f15c00c8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.593+0000 7f15c68cf700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15c00c8a70 0x7f15c01e7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.593+0000 7f15c68cf700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c01e3110 con 0x7f15c00d1010 2023-11-24T06:08:28.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.593+0000 7f15c68cf700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c00d1490 con 0x7f15c01e2d30 2023-11-24T06:08:28.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.593+0000 7f15c68cf700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c00c5c00 con 0x7f15c00c8a70 2023-11-24T06:08:28.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.594+0000 7f15bf7fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15c00d1010 0x7f15c00c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.594+0000 7f15bf7fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15c00d1010 0x7f15c00c8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58706/0 (socket says 172.21.15.91:58706) 2023-11-24T06:08:28.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.594+0000 7f15bf7fe700 1 -- 172.21.15.91:0/1507735387 learned_addr learned my addr 172.21.15.91:0/1507735387 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:28.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.594+0000 7f15c4e6c700 1 --2- 172.21.15.91:0/1507735387 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15c00c8a70 0x7f15c01e7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.594+0000 7f15bffff700 1 --2- 172.21.15.91:0/1507735387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c01e2d30 0x7f15c00d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.594+0000 7f15bf7fe700 1 -- 172.21.15.91:0/1507735387 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15c00c8a70 msgr2=0x7f15c01e7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.594+0000 7f15bf7fe700 1 --2- 172.21.15.91:0/1507735387 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15c00c8a70 0x7f15c01e7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.594+0000 7f15bf7fe700 1 -- 172.21.15.91:0/1507735387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c01e2d30 msgr2=0x7f15c00d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.594+0000 7f15bf7fe700 1 --2- 172.21.15.91:0/1507735387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c01e2d30 0x7f15c00d0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.594+0000 7f15bf7fe700 1 -- 172.21.15.91:0/1507735387 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c00c5ee0 con 0x7f15c00d1010 2023-11-24T06:08:28.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.594+0000 7f15bf7fe700 1 --2- 172.21.15.91:0/1507735387 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15c00d1010 0x7f15c00c8530 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f15b0000fb0 tx=0x7f15b00075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:28.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.594+0000 7f15bcff9700 1 -- 172.21.15.91:0/1507735387 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b000f630 con 0x7f15c00d1010 2023-11-24T06:08:28.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.594+0000 7f15bcff9700 1 -- 172.21.15.91:0/1507735387 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15b000f7d0 con 0x7f15c00d1010 2023-11-24T06:08:28.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.594+0000 7f15c68cf700 1 -- 172.21.15.91:0/1507735387 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c00c61c0 con 0x7f15c00d1010 2023-11-24T06:08:28.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.594+0000 7f15bcff9700 1 -- 172.21.15.91:0/1507735387 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15b000fb80 con 0x7f15c00d1010 2023-11-24T06:08:28.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.594+0000 7f15c68cf700 1 -- 172.21.15.91:0/1507735387 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c00c6780 con 0x7f15c00d1010 2023-11-24T06:08:28.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.595+0000 7f15c68cf700 1 -- 172.21.15.91:0/1507735387 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c00c6910 con 0x7f15c00d1010 2023-11-24T06:08:28.602 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.596+0000 7f15bcff9700 1 -- 172.21.15.91:0/1507735387 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f15b000fce0 con 0x7f15c00d1010 2023-11-24T06:08:28.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.596+0000 7f15bcff9700 1 --2- 172.21.15.91:0/1507735387 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15a006aed0 0x7f15a006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.596+0000 7f15bcff9700 1 -- 172.21.15.91:0/1507735387 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15b0089450 con 0x7f15c00d1010 2023-11-24T06:08:28.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.596+0000 7f15bffff700 1 --2- 172.21.15.91:0/1507735387 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15a006aed0 0x7f15a006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.596+0000 7f15bffff700 1 --2- 172.21.15.91:0/1507735387 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15a006aed0 0x7f15a006d390 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f15a8009c60 tx=0x7f15a8009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:28.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.600+0000 7f15bcff9700 1 -- 172.21.15.91:0/1507735387 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15b0054590 con 0x7f15c00d1010 2023-11-24T06:08:28.794 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.791+0000 7f15c68cf700 1 -- 172.21.15.91:0/1507735387 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f15c0075480 con 0x7f15c00d1010 2023-11-24T06:08:28.794 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.791+0000 7f15bcff9700 1 -- 172.21.15.91:0/1507735387 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f15b0057bb0 con 0x7f15c00d1010 2023-11-24T06:08:28.795 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 64 2023-11-24T06:08:28.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.793+0000 7f15c68cf700 1 -- 172.21.15.91:0/1507735387 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15a006aed0 msgr2=0x7f15a006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.793+0000 7f15c68cf700 1 --2- 172.21.15.91:0/1507735387 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15a006aed0 0x7f15a006d390 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f15a8009c60 tx=0x7f15a8009210 comp rx=0 tx=0).stop 2023-11-24T06:08:28.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.793+0000 7f15c68cf700 1 -- 172.21.15.91:0/1507735387 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15c00d1010 msgr2=0x7f15c00c8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.794+0000 7f15c68cf700 1 --2- 172.21.15.91:0/1507735387 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15c00d1010 0x7f15c00c8530 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f15b0000fb0 tx=0x7f15b00075f0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.794+0000 7f15c68cf700 1 -- 172.21.15.91:0/1507735387 shutdown_connections 2023-11-24T06:08:28.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.794+0000 7f15c68cf700 1 --2- 172.21.15.91:0/1507735387 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15a006aed0 0x7f15a006d390 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.794+0000 7f15c68cf700 1 --2- 172.21.15.91:0/1507735387 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15c01e2d30 0x7f15c00d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.794+0000 7f15c68cf700 1 --2- 172.21.15.91:0/1507735387 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15c00c8a70 0x7f15c01e7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.794+0000 7f15c68cf700 1 --2- 172.21.15.91:0/1507735387 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15c00d1010 0x7f15c00c8530 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.794+0000 7f15c68cf700 1 -- 172.21.15.91:0/1507735387 >> 172.21.15.91:0/1507735387 conn(0x7f15c00b2740 msgr2=0x7f15c0147260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:28.798 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.794+0000 7f15c68cf700 1 -- 172.21.15.91:0/1507735387 shutdown_connections 2023-11-24T06:08:28.798 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.794+0000 7f15c68cf700 1 -- 172.21.15.91:0/1507735387 wait complete. 2023-11-24T06:08:28.808 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:28.928 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.925+0000 7f0d0d807700 1 Processor -- start 2023-11-24T06:08:28.929 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.926+0000 7f0d0d807700 1 -- start start 2023-11-24T06:08:28.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.926+0000 7f0d0d807700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0d0806b620 0x7f0d080b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.926+0000 7f0d0d807700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d081629c0 con 0x7f0d0806ee60 2023-11-24T06:08:28.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.926+0000 7f0d0d807700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d08071f50 con 0x7f0d080b5ab0 2023-11-24T06:08:28.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.926+0000 7f0d0d807700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d08072230 con 0x7f0d0806b620 2023-11-24T06:08:28.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.926+0000 7f0d077fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f0d0806ee60 0x7f0d081622b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53650/0 (socket says 172.21.15.91:53650) 2023-11-24T06:08:28.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.926+0000 7f0d077fe700 1 -- 172.21.15.91:0/3510738870 learned_addr learned my addr 172.21.15.91:0/3510738870 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:28.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d05ffb700 1 -- 172.21.15.91:0/3510738870 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599954058 0 0) 0x7f0d081629c0 con 0x7f0d0806ee60 2023-11-24T06:08:28.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d06ffd700 1 --2- 172.21.15.91:0/3510738870 >> v2:172.21.15.94:3300/0 conn(0x7f0d0806b620 0x7f0d080b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d05ffb700 1 -- 172.21.15.91:0/3510738870 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf00035b0 con 0x7f0d0806ee60 2023-11-24T06:08:28.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d05ffb700 1 -- 172.21.15.91:0/3510738870 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 34156816 0 0) 0x7f0cf00035b0 con 0x7f0d0806ee60 2023-11-24T06:08:28.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d05ffb700 1 -- 172.21.15.91:0/3510738870 >> v2:172.21.15.94:3300/0 conn(0x7f0d0806b620 msgr2=0x7f0d080b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:28.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d05ffb700 1 --2- 172.21.15.91:0/3510738870 >> v2:172.21.15.94:3300/0 conn(0x7f0d0806b620 0x7f0d080b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d05ffb700 1 -- 172.21.15.91:0/3510738870 >> v1:172.21.15.94:6789/0 conn(0x7f0d080b5ab0 legacy=0x7f0d08161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d05ffb700 1 -- 172.21.15.91:0/3510738870 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d08072510 con 0x7f0d0806ee60 2023-11-24T06:08:28.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d05ffb700 1 -- 172.21.15.91:0/3510738870 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f0d000019f0 con 0x7f0d0806ee60 2023-11-24T06:08:28.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d05ffb700 1 -- 172.21.15.91:0/3510738870 >> v1:172.21.15.91:6789/0 conn(0x7f0d0806ee60 legacy=0x7f0d081622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d05ffb700 1 --2- 172.21.15.91:0/3510738870 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0cf0003a00 0x7f0cf0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d05ffb700 1 --2- 172.21.15.91:0/3510738870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cf0004850 0x7f0cf0004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d05ffb700 1 --2- 172.21.15.91:0/3510738870 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0cf0005210 0x7f0cf000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d05ffb700 1 -- 172.21.15.91:0/3510738870 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d08072230 con 0x7f0cf0003a00 2023-11-24T06:08:28.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d05ffb700 1 -- 172.21.15.91:0/3510738870 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf000c160 con 0x7f0cf0004850 2023-11-24T06:08:28.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d05ffb700 1 -- 172.21.15.91:0/3510738870 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf000c320 con 0x7f0cf0005210 2023-11-24T06:08:28.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d06ffd700 1 --2- 172.21.15.91:0/3510738870 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0cf0003a00 0x7f0cf0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d067fc700 1 --2- 172.21.15.91:0/3510738870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cf0004850 0x7f0cf0004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d077fe700 1 --2- 172.21.15.91:0/3510738870 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0cf0005210 0x7f0cf000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.933 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d06ffd700 1 -- 172.21.15.91:0/3510738870 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0cf0005210 msgr2=0x7f0cf000ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:28.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d06ffd700 1 --2- 172.21.15.91:0/3510738870 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0cf0005210 0x7f0cf000ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d06ffd700 1 -- 172.21.15.91:0/3510738870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cf0004850 msgr2=0x7f0cf0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:28.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d06ffd700 1 --2- 172.21.15.91:0/3510738870 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cf0004850 0x7f0cf0004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d06ffd700 1 -- 172.21.15.91:0/3510738870 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf00035b0 con 0x7f0cf0003a00 2023-11-24T06:08:28.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d077fe700 1 --2- 172.21.15.91:0/3510738870 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0cf0005210 0x7f0cf000ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:28.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.927+0000 7f0d077fe700 1 -- 172.21.15.91:0/3510738870 reap_dead start 2023-11-24T06:08:28.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.928+0000 7f0d06ffd700 1 --2- 172.21.15.91:0/3510738870 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0cf0003a00 0x7f0cf0003e20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc009890 tx=0x7f0cf0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa7a4cb75c9ecf9a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:28.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.928+0000 7f0d05ffb700 1 -- 172.21.15.91:0/3510738870 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc00c070 con 0x7f0cf0003a00 2023-11-24T06:08:28.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.928+0000 7f0d05ffb700 1 -- 172.21.15.91:0/3510738870 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc013610 con 0x7f0cf0003a00 2023-11-24T06:08:28.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.928+0000 7f0d05ffb700 1 -- 172.21.15.91:0/3510738870 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc0139a0 con 0x7f0cf0003a00 2023-11-24T06:08:28.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.928+0000 7f0d0d807700 1 -- 172.21.15.91:0/3510738870 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0cf0003a00 msgr2=0x7f0cf0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.928+0000 7f0d0d807700 1 --2- 172.21.15.91:0/3510738870 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0cf0003a00 0x7f0cf0003e20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc009890 tx=0x7f0cf0005880 comp rx=0 tx=0).stop 2023-11-24T06:08:28.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.928+0000 7f0d0d807700 1 -- 172.21.15.91:0/3510738870 shutdown_connections 2023-11-24T06:08:28.934 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.928+0000 7f0d0d807700 1 --2- 172.21.15.91:0/3510738870 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0cf0003a00 0x7f0cf0003e20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.935 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.928+0000 7f0d0d807700 1 -- 172.21.15.91:0/3510738870 >> 172.21.15.91:0/3510738870 conn(0x7f0d080b2740 msgr2=0x7f0d080b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:28.935 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.928+0000 7f0d0d807700 1 -- 172.21.15.91:0/3510738870 shutdown_connections 2023-11-24T06:08:28.935 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.928+0000 7f0d0d807700 1 -- 172.21.15.91:0/3510738870 wait complete. 2023-11-24T06:08:28.937 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.929+0000 7f0d0d807700 1 Processor -- start 2023-11-24T06:08:28.937 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.930+0000 7f0d0d807700 1 -- start start 2023-11-24T06:08:28.937 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.930+0000 7f0d0d807700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0cf0003a00 0x7f0d080cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.937 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.930+0000 7f0d0d807700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d08151980 0x7f0d080c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.937 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.930+0000 7f0d0d807700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0d08140d10 0x7f0d081411a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.937 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.930+0000 7f0d0d807700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d08161e30 con 0x7f0d08140d10 2023-11-24T06:08:28.937 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.930+0000 7f0d0d807700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d08152a20 con 0x7f0d08151980 2023-11-24T06:08:28.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.930+0000 7f0d0d807700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d08151e10 con 0x7f0cf0003a00 2023-11-24T06:08:28.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.930+0000 7f0d077fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0d08140d10 0x7f0d081411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.930+0000 7f0d077fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0d08140d10 0x7f0d081411a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58728/0 (socket says 172.21.15.91:58728) 2023-11-24T06:08:28.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.930+0000 7f0d077fe700 1 -- 172.21.15.91:0/3829570650 learned_addr learned my addr 172.21.15.91:0/3829570650 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:28.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.931+0000 7f0d06ffd700 1 --2- 172.21.15.91:0/3829570650 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0cf0003a00 0x7f0d080cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.931+0000 7f0d067fc700 1 --2- 172.21.15.91:0/3829570650 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d08151980 0x7f0d080c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.931+0000 7f0d077fe700 1 -- 172.21.15.91:0/3829570650 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0cf0003a00 msgr2=0x7f0d080cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.931+0000 7f0d077fe700 1 --2- 172.21.15.91:0/3829570650 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0cf0003a00 0x7f0d080cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.931+0000 7f0d077fe700 1 -- 172.21.15.91:0/3829570650 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d08151980 msgr2=0x7f0d080c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:28.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.931+0000 7f0d077fe700 1 --2- 172.21.15.91:0/3829570650 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d08151980 0x7f0d080c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:28.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.931+0000 7f0d077fe700 1 -- 172.21.15.91:0/3829570650 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d081520f0 con 0x7f0d08140d10 2023-11-24T06:08:28.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.931+0000 7f0d06ffd700 1 --2- 172.21.15.91:0/3829570650 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0cf0003a00 0x7f0d080cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:28.938 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.931+0000 7f0d067fc700 1 --2- 172.21.15.91:0/3829570650 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d08151980 0x7f0d080c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:28.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.931+0000 7f0d077fe700 1 --2- 172.21.15.91:0/3829570650 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0d08140d10 0x7f0d081411a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0d08062d30 tx=0x7f0d0000e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:28.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.931+0000 7f0ceffff700 1 -- 172.21.15.91:0/3829570650 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0000ecd0 con 0x7f0d08140d10 2023-11-24T06:08:28.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.931+0000 7f0ceffff700 1 -- 172.21.15.91:0/3829570650 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0000ee50 con 0x7f0d08140d10 2023-11-24T06:08:28.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.931+0000 7f0d0d807700 1 -- 172.21.15.91:0/3829570650 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d081523d0 con 0x7f0d08140d10 2023-11-24T06:08:28.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.931+0000 7f0ceffff700 1 -- 172.21.15.91:0/3829570650 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d00013660 con 0x7f0d08140d10 2023-11-24T06:08:28.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.931+0000 7f0d0d807700 1 -- 172.21.15.91:0/3829570650 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d080d0050 con 0x7f0d08140d10 2023-11-24T06:08:28.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.932+0000 7f0d0d807700 1 -- 172.21.15.91:0/3829570650 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d08067a40 con 0x7f0d08140d10 2023-11-24T06:08:28.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.933+0000 7f0ceffff700 1 -- 172.21.15.91:0/3829570650 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0d000137c0 con 0x7f0d08140d10 2023-11-24T06:08:28.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.933+0000 7f0ceffff700 1 --2- 172.21.15.91:0/3829570650 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0cf406aed0 0x7f0cf406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:28.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.933+0000 7f0ceffff700 1 -- 172.21.15.91:0/3829570650 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d00089c70 con 0x7f0d08140d10 2023-11-24T06:08:28.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.933+0000 7f0d06ffd700 1 --2- 172.21.15.91:0/3829570650 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0cf406aed0 0x7f0cf406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:28.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.933+0000 7f0d06ffd700 1 --2- 172.21.15.91:0/3829570650 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0cf406aed0 0x7f0cf406d390 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc00a010 tx=0x7f0cfc00c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:28.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:28.937+0000 7f0ceffff700 1 -- 172.21.15.91:0/3829570650 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d08067a40 con 0x7f0d08140d10 2023-11-24T06:08:29.131 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:29.128+0000 7f0d0d807700 1 -- 172.21.15.91:0/3829570650 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d08067a40 con 0x7f0d08140d10 2023-11-24T06:08:31.594 DEBUG:teuthology.orchestra.run.smithi091:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:08:31.594 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:29.129+0000 7f0ceffff700 1 -- 172.21.15.91:0/3829570650 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7f0d08067a40 con 0x7f0d08140d10 2023-11-24T06:08:31.595 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:31.595 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":20,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:02.415612+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:31.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:29.131+0000 7f0d0d807700 1 -- 172.21.15.91:0/3829570650 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0cf406aed0 msgr2=0x7f0cf406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:31.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:29.131+0000 7f0d0d807700 1 --2- 172.21.15.91:0/3829570650 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0cf406aed0 0x7f0cf406d390 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc00a010 tx=0x7f0cfc00c000 comp rx=0 tx=0).stop 2023-11-24T06:08:31.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:29.131+0000 7f0d0d807700 1 -- 172.21.15.91:0/3829570650 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0d08140d10 msgr2=0x7f0d081411a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:31.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:29.131+0000 7f0d0d807700 1 --2- 172.21.15.91:0/3829570650 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0d08140d10 0x7f0d081411a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0d08062d30 tx=0x7f0d0000e440 comp rx=0 tx=0).stop 2023-11-24T06:08:31.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:29.131+0000 7f0d0d807700 1 -- 172.21.15.91:0/3829570650 shutdown_connections 2023-11-24T06:08:31.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:29.131+0000 7f0d0d807700 1 --2- 172.21.15.91:0/3829570650 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0cf406aed0 0x7f0cf406d390 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:31.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:29.131+0000 7f0d0d807700 1 --2- 172.21.15.91:0/3829570650 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0d08140d10 0x7f0d081411a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:31.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:29.131+0000 7f0d0d807700 1 --2- 172.21.15.91:0/3829570650 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0cf0003a00 0x7f0d080cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:31.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:29.131+0000 7f0d0d807700 1 --2- 172.21.15.91:0/3829570650 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0d08151980 0x7f0d080c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:31.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:29.131+0000 7f0d0d807700 1 -- 172.21.15.91:0/3829570650 >> 172.21.15.91:0/3829570650 conn(0x7f0d080b2740 msgr2=0x7f0d080b3830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:31.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:29.132+0000 7f0d0d807700 1 -- 172.21.15.91:0/3829570650 shutdown_connections 2023-11-24T06:08:31.597 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:29.132+0000 7f0d0d807700 1 -- 172.21.15.91:0/3829570650 wait complete. 2023-11-24T06:08:31.598 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:07:58.710106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:07:59.774819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:08:31.599 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:08:31.639 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:08:31.775 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.772+0000 7f13abfc2700 1 Processor -- start 2023-11-24T06:08:31.775 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.772+0000 7f13abfc2700 1 -- start start 2023-11-24T06:08:31.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.773+0000 7f13abfc2700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f13a415add0 0x7f13a40b4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:31.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.773+0000 7f13abfc2700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f13a406f540 0x7f13a40cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:31.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.773+0000 7f13abfc2700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a40ce5b0 con 0x7f13a40b3ab0 2023-11-24T06:08:31.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.773+0000 7f13abfc2700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a4072d70 con 0x7f13a406f540 2023-11-24T06:08:31.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.773+0000 7f13abfc2700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a40b4f10 con 0x7f13a415add0 2023-11-24T06:08:31.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.773+0000 7f13aa55f700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f13a406f540 0x7f13a40cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:31.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.773+0000 7f13a9d5e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f13a415add0 0x7f13a40b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:31.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.773+0000 7f13a9d5e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f13a415add0 0x7f13a40b4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:42164/0 (socket says 172.21.15.91:42164) 2023-11-24T06:08:31.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.773+0000 7f13a9d5e700 1 -- 172.21.15.91:0/3637339754 learned_addr learned my addr 172.21.15.91:0/3637339754 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:31.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.773+0000 7f13aa55f700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f13a406f540 0x7f13a40cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:43128/0 (socket says 172.21.15.91:43128) 2023-11-24T06:08:31.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.773+0000 7f13aa55f700 1 -- 172.21.15.91:0/3637339754 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f13a415add0 msgr2=0x7f13a40b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:31.778 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.773+0000 7f13aa55f700 1 --2- 172.21.15.91:0/3637339754 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f13a415add0 0x7f13a40b4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:31.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.773+0000 7f13aa55f700 1 -- 172.21.15.91:0/3637339754 >> v1:172.21.15.94:6789/0 conn(0x7f13a40b3ab0 legacy=0x7f13a406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:31.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.773+0000 7f13aa55f700 1 -- 172.21.15.91:0/3637339754 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a40b51f0 con 0x7f13a406f540 2023-11-24T06:08:31.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.773+0000 7f13a9d5e700 1 --2- 172.21.15.91:0/3637339754 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f13a415add0 0x7f13a40b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:08:31.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.774+0000 7f13aa55f700 1 --2- 172.21.15.91:0/3637339754 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13a406f540 0x7f13a40cea40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f139c009d90 tx=0x7f139c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b25c62f7555c0630 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:31.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.774+0000 7f13a8d5c700 1 -- 172.21.15.91:0/3637339754 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c00e070 con 0x7f13a406f540 2023-11-24T06:08:31.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.774+0000 7f13a8d5c700 1 -- 172.21.15.91:0/3637339754 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139c00bb20 con 0x7f13a406f540 2023-11-24T06:08:31.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.774+0000 7f13a8d5c700 1 -- 172.21.15.91:0/3637339754 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c012430 con 0x7f13a406f540 2023-11-24T06:08:31.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.774+0000 7f13abfc2700 1 -- 172.21.15.91:0/3637339754 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13a406f540 msgr2=0x7f13a40cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:31.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.774+0000 7f13abfc2700 1 --2- 172.21.15.91:0/3637339754 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13a406f540 0x7f13a40cea40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f139c009d90 tx=0x7f139c00b330 comp rx=0 tx=0).stop 2023-11-24T06:08:31.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.774+0000 7f13abfc2700 1 -- 172.21.15.91:0/3637339754 shutdown_connections 2023-11-24T06:08:31.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.774+0000 7f13abfc2700 1 --2- 172.21.15.91:0/3637339754 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13a406f540 0x7f13a40cea40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:31.779 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.774+0000 7f13abfc2700 1 --2- 172.21.15.91:0/3637339754 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f13a415add0 0x7f13a40b4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:31.780 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.774+0000 7f13abfc2700 1 -- 172.21.15.91:0/3637339754 >> 172.21.15.91:0/3637339754 conn(0x7f13a40b27b0 msgr2=0x7f13a40b2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:31.780 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.774+0000 7f13abfc2700 1 -- 172.21.15.91:0/3637339754 shutdown_connections 2023-11-24T06:08:31.780 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.774+0000 7f13abfc2700 1 -- 172.21.15.91:0/3637339754 wait complete. 2023-11-24T06:08:31.782 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.776+0000 7f13abfc2700 1 Processor -- start 2023-11-24T06:08:31.782 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.776+0000 7f13abfc2700 1 -- start start 2023-11-24T06:08:31.783 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.776+0000 7f13abfc2700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f13a406f540 0x7f13a40cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:31.783 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.776+0000 7f13abfc2700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13a40b3ab0 0x7f13a415dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:31.783 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.776+0000 7f13abfc2700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f13a415add0 0x7f13a414ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:31.783 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.776+0000 7f13abfc2700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a41f2970 con 0x7f13a406f540 2023-11-24T06:08:31.783 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.776+0000 7f13abfc2700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a41f2c50 con 0x7f13a40b3ab0 2023-11-24T06:08:31.783 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.776+0000 7f13abfc2700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a41f2f30 con 0x7f13a415add0 2023-11-24T06:08:31.783 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.776+0000 7f13a9d5e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f13a406f540 0x7f13a40cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:31.783 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.776+0000 7f13a9d5e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f13a406f540 0x7f13a40cfd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58738/0 (socket says 172.21.15.91:58738) 2023-11-24T06:08:31.783 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.777+0000 7f13a9d5e700 1 -- 172.21.15.91:0/1044440871 learned_addr learned my addr 172.21.15.91:0/1044440871 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:31.783 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.777+0000 7f13aa55f700 1 --2- 172.21.15.91:0/1044440871 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f13a415add0 0x7f13a414ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:31.783 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.777+0000 7f13a955d700 1 --2- 172.21.15.91:0/1044440871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13a40b3ab0 0x7f13a415dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:31.783 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.777+0000 7f13a9d5e700 1 -- 172.21.15.91:0/1044440871 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f13a415add0 msgr2=0x7f13a414ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:31.784 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.777+0000 7f13a9d5e700 1 --2- 172.21.15.91:0/1044440871 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f13a415add0 0x7f13a414ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:31.784 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.777+0000 7f13a9d5e700 1 -- 172.21.15.91:0/1044440871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13a40b3ab0 msgr2=0x7f13a415dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:31.784 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.777+0000 7f13a9d5e700 1 --2- 172.21.15.91:0/1044440871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13a40b3ab0 0x7f13a415dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:31.784 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.777+0000 7f13a9d5e700 1 -- 172.21.15.91:0/1044440871 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a0009550 con 0x7f13a406f540 2023-11-24T06:08:31.784 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.777+0000 7f13a9d5e700 1 --2- 172.21.15.91:0/1044440871 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f13a406f540 0x7f13a40cfd60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f13a000e9b0 tx=0x7f13a000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:31.784 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.777+0000 7f139a7fc700 1 -- 172.21.15.91:0/1044440871 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a000c850 con 0x7f13a406f540 2023-11-24T06:08:31.784 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.777+0000 7f139a7fc700 1 -- 172.21.15.91:0/1044440871 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a000c9b0 con 0x7f13a406f540 2023-11-24T06:08:31.784 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.777+0000 7f13abfc2700 1 -- 172.21.15.91:0/1044440871 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139c009a70 con 0x7f13a406f540 2023-11-24T06:08:31.784 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.777+0000 7f139a7fc700 1 -- 172.21.15.91:0/1044440871 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a000cd20 con 0x7f13a406f540 2023-11-24T06:08:31.784 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.777+0000 7f13abfc2700 1 -- 172.21.15.91:0/1044440871 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a41f34f0 con 0x7f13a406f540 2023-11-24T06:08:31.784 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.778+0000 7f13abfc2700 1 -- 172.21.15.91:0/1044440871 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a4075480 con 0x7f13a406f540 2023-11-24T06:08:31.784 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.778+0000 7f139a7fc700 1 -- 172.21.15.91:0/1044440871 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f13a0013470 con 0x7f13a406f540 2023-11-24T06:08:31.785 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.779+0000 7f139a7fc700 1 --2- 172.21.15.91:0/1044440871 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f138806aed0 0x7f138806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:31.785 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.779+0000 7f139a7fc700 1 -- 172.21.15.91:0/1044440871 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13a0089710 con 0x7f13a406f540 2023-11-24T06:08:31.785 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.779+0000 7f13a955d700 1 --2- 172.21.15.91:0/1044440871 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f138806aed0 0x7f138806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:31.785 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.779+0000 7f13a955d700 1 --2- 172.21.15.91:0/1044440871 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f138806aed0 0x7f138806d390 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1394000f60 tx=0x7f1394009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:31.786 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.783+0000 7f139a7fc700 1 -- 172.21.15.91:0/1044440871 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13a00547d0 con 0x7f13a406f540 2023-11-24T06:08:31.975 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.972+0000 7f13abfc2700 1 -- 172.21.15.91:0/1044440871 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f13a406b100 con 0x7f13a406f540 2023-11-24T06:08:31.975 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.973+0000 7f139a7fc700 1 -- 172.21.15.91:0/1044440871 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f13a0057df0 con 0x7f13a406f540 2023-11-24T06:08:31.976 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:08:31.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.975+0000 7f13abfc2700 1 -- 172.21.15.91:0/1044440871 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f138806aed0 msgr2=0x7f138806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:31.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.975+0000 7f13abfc2700 1 --2- 172.21.15.91:0/1044440871 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f138806aed0 0x7f138806d390 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1394000f60 tx=0x7f1394009040 comp rx=0 tx=0).stop 2023-11-24T06:08:31.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.975+0000 7f13abfc2700 1 -- 172.21.15.91:0/1044440871 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f13a406f540 msgr2=0x7f13a40cfd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:31.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.975+0000 7f13abfc2700 1 --2- 172.21.15.91:0/1044440871 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f13a406f540 0x7f13a40cfd60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f13a000e9b0 tx=0x7f13a000ed70 comp rx=0 tx=0).stop 2023-11-24T06:08:31.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.975+0000 7f13abfc2700 1 -- 172.21.15.91:0/1044440871 shutdown_connections 2023-11-24T06:08:31.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.975+0000 7f13abfc2700 1 --2- 172.21.15.91:0/1044440871 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f138806aed0 0x7f138806d390 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:31.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.975+0000 7f13abfc2700 1 --2- 172.21.15.91:0/1044440871 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f13a415add0 0x7f13a414ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:31.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.975+0000 7f13abfc2700 1 --2- 172.21.15.91:0/1044440871 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f13a406f540 0x7f13a40cfd60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:31.979 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.975+0000 7f13abfc2700 1 --2- 172.21.15.91:0/1044440871 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f13a40b3ab0 0x7f13a415dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:31.979 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.975+0000 7f13abfc2700 1 -- 172.21.15.91:0/1044440871 >> 172.21.15.91:0/1044440871 conn(0x7f13a40b27b0 msgr2=0x7f13a40cecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:31.979 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.975+0000 7f13abfc2700 1 -- 172.21.15.91:0/1044440871 shutdown_connections 2023-11-24T06:08:31.979 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:31.975+0000 7f13abfc2700 1 -- 172.21.15.91:0/1044440871 wait complete. 2023-11-24T06:08:31.990 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:08:32.113 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.110+0000 7ff98c4a6700 1 Processor -- start 2023-11-24T06:08:32.113 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.110+0000 7ff98c4a6700 1 -- start start 2023-11-24T06:08:32.114 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.111+0000 7ff98c4a6700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff98406bed0 0x7ff984066ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.114 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.111+0000 7ff98c4a6700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9840b10c0 con 0x7ff9840af980 2023-11-24T06:08:32.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.111+0000 7ff98c4a6700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9840af730 con 0x7ff98415bf10 2023-11-24T06:08:32.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.111+0000 7ff98c4a6700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9840b1c60 con 0x7ff98406bed0 2023-11-24T06:08:32.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.111+0000 7ff98aa43700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff98406bed0 0x7ff984066ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.111+0000 7ff98aa43700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff98406bed0 0x7ff984066ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58750/0 (socket says 172.21.15.91:58750) 2023-11-24T06:08:32.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.111+0000 7ff989a41700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7ff9840af980 0x7ff98406b7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53682/0 (socket says 172.21.15.91:53682) 2023-11-24T06:08:32.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.111+0000 7ff98aa43700 1 -- 172.21.15.91:0/63388789 learned_addr learned my addr 172.21.15.91:0/63388789 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:32.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.111+0000 7ff98aa43700 1 -- 172.21.15.91:0/63388789 >> v1:172.21.15.94:6789/0 conn(0x7ff98415bf10 legacy=0x7ff9840b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.111+0000 7ff98aa43700 1 -- 172.21.15.91:0/63388789 >> v1:172.21.15.91:6789/0 conn(0x7ff9840af980 legacy=0x7ff98406b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.111+0000 7ff98aa43700 1 -- 172.21.15.91:0/63388789 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9840b5ee0 con 0x7ff98406bed0 2023-11-24T06:08:32.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.111+0000 7ff989240700 1 -- 172.21.15.91:0/63388789 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898133910 0 0) 0x7ff9840b10c0 con 0x7ff9840af980 2023-11-24T06:08:32.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.112+0000 7ff98aa43700 1 --2- 172.21.15.91:0/63388789 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff98406bed0 0x7ff984066ce0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009e50 tx=0x7ff97c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f3ff51242e6b726 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:32.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.112+0000 7ff989240700 1 -- 172.21.15.91:0/63388789 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97c00e070 con 0x7ff98406bed0 2023-11-24T06:08:32.115 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.112+0000 7ff989240700 1 -- 172.21.15.91:0/63388789 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97c00b9c0 con 0x7ff98406bed0 2023-11-24T06:08:32.116 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.112+0000 7ff989240700 1 -- 172.21.15.91:0/63388789 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97c00bd70 con 0x7ff98406bed0 2023-11-24T06:08:32.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.112+0000 7ff98c4a6700 1 -- 172.21.15.91:0/63388789 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff98406bed0 msgr2=0x7ff984066ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.112+0000 7ff98c4a6700 1 --2- 172.21.15.91:0/63388789 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff98406bed0 0x7ff984066ce0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009e50 tx=0x7ff97c00b330 comp rx=0 tx=0).stop 2023-11-24T06:08:32.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.112+0000 7ff98c4a6700 1 -- 172.21.15.91:0/63388789 shutdown_connections 2023-11-24T06:08:32.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.112+0000 7ff98c4a6700 1 --2- 172.21.15.91:0/63388789 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff98406bed0 0x7ff984066ce0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.112+0000 7ff98c4a6700 1 -- 172.21.15.91:0/63388789 >> 172.21.15.91:0/63388789 conn(0x7ff9840b5070 msgr2=0x7ff9840b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:32.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.112+0000 7ff98c4a6700 1 -- 172.21.15.91:0/63388789 shutdown_connections 2023-11-24T06:08:32.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.112+0000 7ff98c4a6700 1 -- 172.21.15.91:0/63388789 wait complete. 2023-11-24T06:08:32.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.113+0000 7ff98c4a6700 1 Processor -- start 2023-11-24T06:08:32.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.114+0000 7ff98c4a6700 1 -- start start 2023-11-24T06:08:32.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.114+0000 7ff98c4a6700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff98406bed0 0x7ff984156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.114+0000 7ff98c4a6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff9840af980 0x7ff9841543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.114+0000 7ff98c4a6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff98415bf10 0x7ff984154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.114+0000 7ff98c4a6700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff984154e60 con 0x7ff98406bed0 2023-11-24T06:08:32.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.114+0000 7ff98c4a6700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98415ef90 con 0x7ff9840af980 2023-11-24T06:08:32.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.114+0000 7ff98c4a6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9840c9710 con 0x7ff98415bf10 2023-11-24T06:08:32.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.114+0000 7ff98a242700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff98406bed0 0x7ff984156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.114+0000 7ff98a242700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff98406bed0 0x7ff984156900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58752/0 (socket says 172.21.15.91:58752) 2023-11-24T06:08:32.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.114+0000 7ff98a242700 1 -- 172.21.15.91:0/1158978838 learned_addr learned my addr 172.21.15.91:0/1158978838 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:32.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.115+0000 7ff989a41700 1 --2- 172.21.15.91:0/1158978838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff9840af980 0x7ff9841543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.115+0000 7ff98aa43700 1 --2- 172.21.15.91:0/1158978838 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff98415bf10 0x7ff984154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.115+0000 7ff98a242700 1 -- 172.21.15.91:0/1158978838 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff98415bf10 msgr2=0x7ff984154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.115+0000 7ff98a242700 1 --2- 172.21.15.91:0/1158978838 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff98415bf10 0x7ff984154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.115+0000 7ff98a242700 1 -- 172.21.15.91:0/1158978838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff9840af980 msgr2=0x7ff9841543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.115+0000 7ff98a242700 1 --2- 172.21.15.91:0/1158978838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff9840af980 0x7ff9841543e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.115+0000 7ff98a242700 1 -- 172.21.15.91:0/1158978838 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97c009b30 con 0x7ff98406bed0 2023-11-24T06:08:32.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.115+0000 7ff98a242700 1 --2- 172.21.15.91:0/1158978838 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff98406bed0 0x7ff984156900 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff97400c450 tx=0x7ff97400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:32.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.115+0000 7ff97affd700 1 -- 172.21.15.91:0/1158978838 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9740052d0 con 0x7ff98406bed0 2023-11-24T06:08:32.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.115+0000 7ff97affd700 1 -- 172.21.15.91:0/1158978838 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff974005430 con 0x7ff98406bed0 2023-11-24T06:08:32.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.115+0000 7ff97affd700 1 -- 172.21.15.91:0/1158978838 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9740038c0 con 0x7ff98406bed0 2023-11-24T06:08:32.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.115+0000 7ff98c4a6700 1 -- 172.21.15.91:0/1158978838 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9840c99f0 con 0x7ff98406bed0 2023-11-24T06:08:32.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.115+0000 7ff98c4a6700 1 -- 172.21.15.91:0/1158978838 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff984156e40 con 0x7ff98406bed0 2023-11-24T06:08:32.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.116+0000 7ff98c4a6700 1 -- 172.21.15.91:0/1158978838 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9840ca040 con 0x7ff98406bed0 2023-11-24T06:08:32.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.116+0000 7ff97affd700 1 -- 172.21.15.91:0/1158978838 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff974003a20 con 0x7ff98406bed0 2023-11-24T06:08:32.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.117+0000 7ff97affd700 1 --2- 172.21.15.91:0/1158978838 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff96806aed0 0x7ff96806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.117+0000 7ff97affd700 1 -- 172.21.15.91:0/1158978838 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff9740889c0 con 0x7ff98406bed0 2023-11-24T06:08:32.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.117+0000 7ff989a41700 1 --2- 172.21.15.91:0/1158978838 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff96806aed0 0x7ff96806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.117+0000 7ff989a41700 1 --2- 172.21.15.91:0/1158978838 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff96806aed0 0x7ff96806d390 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff980000f60 tx=0x7ff980007c40 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:32.124 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.121+0000 7ff97affd700 1 -- 172.21.15.91:0/1158978838 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff9740539e0 con 0x7ff98406bed0 2023-11-24T06:08:32.313 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.310+0000 7ff98c4a6700 1 -- 172.21.15.91:0/1158978838 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff984156fd0 con 0x7ff98406bed0 2023-11-24T06:08:32.313 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.310+0000 7ff97affd700 1 -- 172.21.15.91:0/1158978838 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff974057000 con 0x7ff98406bed0 2023-11-24T06:08:32.314 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 32 2023-11-24T06:08:32.316 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.312+0000 7ff98c4a6700 1 -- 172.21.15.91:0/1158978838 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff96806aed0 msgr2=0x7ff96806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.316 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.312+0000 7ff98c4a6700 1 --2- 172.21.15.91:0/1158978838 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff96806aed0 0x7ff96806d390 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff980000f60 tx=0x7ff980007c40 comp rx=0 tx=0).stop 2023-11-24T06:08:32.316 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.312+0000 7ff98c4a6700 1 -- 172.21.15.91:0/1158978838 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff98406bed0 msgr2=0x7ff984156900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.316 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.312+0000 7ff98c4a6700 1 --2- 172.21.15.91:0/1158978838 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff98406bed0 0x7ff984156900 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff97400c450 tx=0x7ff97400c810 comp rx=0 tx=0).stop 2023-11-24T06:08:32.316 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.312+0000 7ff98c4a6700 1 -- 172.21.15.91:0/1158978838 shutdown_connections 2023-11-24T06:08:32.316 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.312+0000 7ff98c4a6700 1 --2- 172.21.15.91:0/1158978838 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff96806aed0 0x7ff96806d390 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.316 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.312+0000 7ff98c4a6700 1 --2- 172.21.15.91:0/1158978838 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff98415bf10 0x7ff984154920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.316 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.312+0000 7ff98c4a6700 1 --2- 172.21.15.91:0/1158978838 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff98406bed0 0x7ff984156900 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.317 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.312+0000 7ff98c4a6700 1 --2- 172.21.15.91:0/1158978838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff9840af980 0x7ff9841543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.317 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.312+0000 7ff98c4a6700 1 -- 172.21.15.91:0/1158978838 >> 172.21.15.91:0/1158978838 conn(0x7ff9840b5070 msgr2=0x7ff9840b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:32.317 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.313+0000 7ff98c4a6700 1 -- 172.21.15.91:0/1158978838 shutdown_connections 2023-11-24T06:08:32.317 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.313+0000 7ff98c4a6700 1 -- 172.21.15.91:0/1158978838 wait complete. 2023-11-24T06:08:32.327 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:08:32.452 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.449+0000 7ff855a06700 1 Processor -- start 2023-11-24T06:08:32.453 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.449+0000 7ff855a06700 1 -- start start 2023-11-24T06:08:32.455 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.450+0000 7ff855a06700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff8500d0870 0x7ff8500d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.455 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.450+0000 7ff855a06700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8500d49f0 0x7ff8500b6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.455 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.450+0000 7ff855a06700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8500b5370 con 0x7ff8500d0c50 2023-11-24T06:08:32.455 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.450+0000 7ff855a06700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8500d2f80 con 0x7ff8500d0870 2023-11-24T06:08:32.455 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.450+0000 7ff855a06700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8500cff70 con 0x7ff8500d49f0 2023-11-24T06:08:32.455 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.451+0000 7ff84effd700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff8500d0870 0x7ff8500d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.455 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.451+0000 7ff84f7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8500d49f0 0x7ff8500b6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.455 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.451+0000 7ff84f7fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8500d49f0 0x7ff8500b6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:42194/0 (socket says 172.21.15.91:42194) 2023-11-24T06:08:32.455 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.451+0000 7ff84effd700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff8500d0870 0x7ff8500d26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:43152/0 (socket says 172.21.15.91:43152) 2023-11-24T06:08:32.455 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.451+0000 7ff84f7fe700 1 -- 172.21.15.91:0/4263316596 learned_addr learned my addr 172.21.15.91:0/4263316596 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:32.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.451+0000 7ff84e7fc700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7ff8500d0c50 0x7ff8500cf860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.91:55486/0 (socket says 172.21.15.91:55486) 2023-11-24T06:08:32.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.451+0000 7ff84f7fe700 1 -- 172.21.15.91:0/4263316596 >> v2:172.21.15.94:3300/0 conn(0x7ff8500d0870 msgr2=0x7ff8500d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:32.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.451+0000 7ff84f7fe700 1 --2- 172.21.15.91:0/4263316596 >> v2:172.21.15.94:3300/0 conn(0x7ff8500d0870 0x7ff8500d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.451+0000 7ff84f7fe700 1 -- 172.21.15.91:0/4263316596 >> v1:172.21.15.94:6789/0 conn(0x7ff8500d0c50 legacy=0x7ff8500cf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.451+0000 7ff84f7fe700 1 -- 172.21.15.91:0/4263316596 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8500ceae0 con 0x7ff8500d49f0 2023-11-24T06:08:32.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.451+0000 7ff84f7fe700 1 --2- 172.21.15.91:0/4263316596 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8500d49f0 0x7ff8500b6d20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff848009e50 tx=0x7ff84800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2de7c87e60996274 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:32.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.451+0000 7ff84dffb700 1 -- 172.21.15.91:0/4263316596 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84800e070 con 0x7ff8500d49f0 2023-11-24T06:08:32.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.451+0000 7ff84dffb700 1 -- 172.21.15.91:0/4263316596 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84800b910 con 0x7ff8500d49f0 2023-11-24T06:08:32.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.451+0000 7ff84dffb700 1 -- 172.21.15.91:0/4263316596 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84800bcc0 con 0x7ff8500d49f0 2023-11-24T06:08:32.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.452+0000 7ff855a06700 1 -- 172.21.15.91:0/4263316596 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8500d49f0 msgr2=0x7ff8500b6d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.452+0000 7ff855a06700 1 --2- 172.21.15.91:0/4263316596 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8500d49f0 0x7ff8500b6d20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff848009e50 tx=0x7ff84800b330 comp rx=0 tx=0).stop 2023-11-24T06:08:32.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.452+0000 7ff855a06700 1 -- 172.21.15.91:0/4263316596 shutdown_connections 2023-11-24T06:08:32.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.452+0000 7ff855a06700 1 --2- 172.21.15.91:0/4263316596 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8500d49f0 0x7ff8500b6d20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.452+0000 7ff855a06700 1 --2- 172.21.15.91:0/4263316596 >> v2:172.21.15.94:3300/0 conn(0x7ff8500d0870 0x7ff8500d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.452+0000 7ff855a06700 1 -- 172.21.15.91:0/4263316596 >> 172.21.15.91:0/4263316596 conn(0x7ff8500d3ed0 msgr2=0x7ff8500d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:32.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.452+0000 7ff855a06700 1 -- 172.21.15.91:0/4263316596 shutdown_connections 2023-11-24T06:08:32.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.452+0000 7ff855a06700 1 -- 172.21.15.91:0/4263316596 wait complete. 2023-11-24T06:08:32.460 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.453+0000 7ff855a06700 1 Processor -- start 2023-11-24T06:08:32.460 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.453+0000 7ff855a06700 1 -- start start 2023-11-24T06:08:32.460 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff855a06700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8500d0870 0x7ff85006c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.460 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff855a06700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8500d0c50 0x7ff85006cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.460 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff855a06700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8500d49f0 0x7ff8501543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.460 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff855a06700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff850154ff0 con 0x7ff8500d0870 2023-11-24T06:08:32.460 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff855a06700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8501f8350 con 0x7ff8500d49f0 2023-11-24T06:08:32.460 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff855a06700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff850151eb0 con 0x7ff8500d0c50 2023-11-24T06:08:32.460 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff84effd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8500d0870 0x7ff85006c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.460 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff84e7fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8500d0c50 0x7ff85006cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.460 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff84effd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8500d0870 0x7ff85006c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58764/0 (socket says 172.21.15.91:58764) 2023-11-24T06:08:32.461 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff84effd700 1 -- 172.21.15.91:0/659182006 learned_addr learned my addr 172.21.15.91:0/659182006 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:32.461 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff84f7fe700 1 --2- 172.21.15.91:0/659182006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8500d49f0 0x7ff8501543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.461 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff84effd700 1 -- 172.21.15.91:0/659182006 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8500d0c50 msgr2=0x7ff85006cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.461 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff84effd700 1 --2- 172.21.15.91:0/659182006 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8500d0c50 0x7ff85006cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.461 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff84effd700 1 -- 172.21.15.91:0/659182006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8500d49f0 msgr2=0x7ff8501543b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.461 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff84effd700 1 --2- 172.21.15.91:0/659182006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8500d49f0 0x7ff8501543b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.461 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff84effd700 1 -- 172.21.15.91:0/659182006 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff848009b30 con 0x7ff8500d0870 2023-11-24T06:08:32.461 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.454+0000 7ff84e7fc700 1 --2- 172.21.15.91:0/659182006 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8500d0c50 0x7ff85006cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:08:32.461 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.455+0000 7ff84effd700 1 --2- 172.21.15.91:0/659182006 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8500d0870 0x7ff85006c7f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff84400e9b0 tx=0x7ff84400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:32.461 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.455+0000 7ff837fff700 1 -- 172.21.15.91:0/659182006 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844009710 con 0x7ff8500d0870 2023-11-24T06:08:32.461 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.455+0000 7ff837fff700 1 -- 172.21.15.91:0/659182006 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84400ee50 con 0x7ff8500d0870 2023-11-24T06:08:32.461 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.455+0000 7ff855a06700 1 -- 172.21.15.91:0/659182006 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff850152100 con 0x7ff8500d0870 2023-11-24T06:08:32.462 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.455+0000 7ff837fff700 1 -- 172.21.15.91:0/659182006 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84400c9f0 con 0x7ff8500d0870 2023-11-24T06:08:32.462 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.455+0000 7ff855a06700 1 -- 172.21.15.91:0/659182006 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8501526c0 con 0x7ff8500d0870 2023-11-24T06:08:32.462 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.456+0000 7ff855a06700 1 -- 172.21.15.91:0/659182006 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8500b57c0 con 0x7ff8500d0870 2023-11-24T06:08:32.462 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.456+0000 7ff837fff700 1 -- 172.21.15.91:0/659182006 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff84400cb50 con 0x7ff8500d0870 2023-11-24T06:08:32.462 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.457+0000 7ff837fff700 1 --2- 172.21.15.91:0/659182006 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff83c06aed0 0x7ff83c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.462 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.457+0000 7ff837fff700 1 -- 172.21.15.91:0/659182006 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8440894b0 con 0x7ff8500d0870 2023-11-24T06:08:32.462 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.457+0000 7ff84e7fc700 1 --2- 172.21.15.91:0/659182006 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff83c06aed0 0x7ff83c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.462 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.457+0000 7ff84e7fc700 1 --2- 172.21.15.91:0/659182006 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff83c06aed0 0x7ff83c06d390 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff8500b5010 tx=0x7ff840007ed0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:32.464 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.461+0000 7ff837fff700 1 -- 172.21.15.91:0/659182006 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff8440545f0 con 0x7ff8500d0870 2023-11-24T06:08:32.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.653+0000 7ff855a06700 1 -- 172.21.15.91:0/659182006 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff850067a20 con 0x7ff8500d0870 2023-11-24T06:08:32.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.653+0000 7ff837fff700 1 -- 172.21.15.91:0/659182006 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff844057c10 con 0x7ff8500d0870 2023-11-24T06:08:32.656 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 64 2023-11-24T06:08:32.658 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.655+0000 7ff855a06700 1 -- 172.21.15.91:0/659182006 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff83c06aed0 msgr2=0x7ff83c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.658 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.655+0000 7ff855a06700 1 --2- 172.21.15.91:0/659182006 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff83c06aed0 0x7ff83c06d390 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff8500b5010 tx=0x7ff840007ed0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.655+0000 7ff855a06700 1 -- 172.21.15.91:0/659182006 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8500d0870 msgr2=0x7ff85006c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.655+0000 7ff855a06700 1 --2- 172.21.15.91:0/659182006 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8500d0870 0x7ff85006c7f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff84400e9b0 tx=0x7ff84400ecc0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.655+0000 7ff855a06700 1 -- 172.21.15.91:0/659182006 shutdown_connections 2023-11-24T06:08:32.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.655+0000 7ff855a06700 1 --2- 172.21.15.91:0/659182006 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff83c06aed0 0x7ff83c06d390 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.655+0000 7ff855a06700 1 --2- 172.21.15.91:0/659182006 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff8500d49f0 0x7ff8501543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.655+0000 7ff855a06700 1 --2- 172.21.15.91:0/659182006 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff8500d0870 0x7ff85006c7f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.655+0000 7ff855a06700 1 --2- 172.21.15.91:0/659182006 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff8500d0c50 0x7ff85006cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.655+0000 7ff855a06700 1 -- 172.21.15.91:0/659182006 >> 172.21.15.91:0/659182006 conn(0x7ff8500d3ed0 msgr2=0x7ff8500d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:32.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.655+0000 7ff855a06700 1 -- 172.21.15.91:0/659182006 shutdown_connections 2023-11-24T06:08:32.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.655+0000 7ff855a06700 1 -- 172.21.15.91:0/659182006 wait complete. 2023-11-24T06:08:32.744 INFO:tasks.cephfs_test_runner:Starting test: test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) ensure failure status is shown when clone is in failed state and validate the reason 2023-11-24T06:08:32.745 INFO:tasks.cephfs_test_runner:test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T06:08:32.745 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:32.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.863+0000 7fd83a742700 1 Processor -- start 2023-11-24T06:08:32.866 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.863+0000 7fd83a742700 1 -- start start 2023-11-24T06:08:32.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.863+0000 7fd83a742700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd83415e1c0 0x7fd83406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.863+0000 7fd83a742700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd8340b0e90 0x7fd83406f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.863+0000 7fd83a742700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8340d1230 con 0x7fd83406fa80 2023-11-24T06:08:32.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.863+0000 7fd83a742700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd83406ff10 con 0x7fd8340b0e90 2023-11-24T06:08:32.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.863+0000 7fd83a742700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340d0c70 con 0x7fd83415e1c0 2023-11-24T06:08:32.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.863+0000 7fd8337fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd8340b0e90 0x7fd83406f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.863+0000 7fd8337fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd8340b0e90 0x7fd83406f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58772/0 (socket says 172.21.15.91:58772) 2023-11-24T06:08:32.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.863+0000 7fd8337fe700 1 -- 172.21.15.91:0/1708067989 learned_addr learned my addr 172.21.15.91:0/1708067989 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:32.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.863+0000 7fd838cdf700 1 --1- 172.21.15.91:0/1708067989 >> v1:172.21.15.91:6789/0 conn(0x7fd83406fa80 0x7fd834161d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53710/0 (socket says 172.21.15.91:53710) 2023-11-24T06:08:32.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.864+0000 7fd8337fe700 1 -- 172.21.15.91:0/1708067989 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd83415e1c0 msgr2=0x7fd83406eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:32.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.864+0000 7fd833fff700 1 --2- 172.21.15.91:0/1708067989 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd83415e1c0 0x7fd83406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.864+0000 7fd832ffd700 1 -- 172.21.15.91:0/1708067989 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146200508 0 0) 0x7fd8340d1230 con 0x7fd83406fa80 2023-11-24T06:08:32.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.864+0000 7fd8337fe700 1 --2- 172.21.15.91:0/1708067989 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd83415e1c0 0x7fd83406eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.864+0000 7fd8337fe700 1 -- 172.21.15.91:0/1708067989 >> v1:172.21.15.91:6789/0 conn(0x7fd83406fa80 legacy=0x7fd834161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.864+0000 7fd8337fe700 1 -- 172.21.15.91:0/1708067989 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8340d0f50 con 0x7fd8340b0e90 2023-11-24T06:08:32.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.864+0000 7fd8337fe700 1 --2- 172.21.15.91:0/1708067989 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd8340b0e90 0x7fd83406f540 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd8240098f0 tx=0x7fd824009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a50edbac52f4b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:32.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.864+0000 7fd832ffd700 1 -- 172.21.15.91:0/1708067989 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82400b8a0 con 0x7fd8340b0e90 2023-11-24T06:08:32.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.864+0000 7fd832ffd700 1 -- 172.21.15.91:0/1708067989 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82400ba00 con 0x7fd8340b0e90 2023-11-24T06:08:32.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.864+0000 7fd832ffd700 1 -- 172.21.15.91:0/1708067989 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82400bd90 con 0x7fd8340b0e90 2023-11-24T06:08:32.870 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.865+0000 7fd83a742700 1 -- 172.21.15.91:0/1708067989 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd8340b0e90 msgr2=0x7fd83406f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.870 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.865+0000 7fd83a742700 1 --2- 172.21.15.91:0/1708067989 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd8340b0e90 0x7fd83406f540 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd8240098f0 tx=0x7fd824009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:32.870 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.865+0000 7fd83a742700 1 -- 172.21.15.91:0/1708067989 shutdown_connections 2023-11-24T06:08:32.870 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.865+0000 7fd83a742700 1 --2- 172.21.15.91:0/1708067989 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd83415e1c0 0x7fd83406eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.870 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.865+0000 7fd83a742700 1 --2- 172.21.15.91:0/1708067989 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd8340b0e90 0x7fd83406f540 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.870 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.865+0000 7fd83a742700 1 -- 172.21.15.91:0/1708067989 >> 172.21.15.91:0/1708067989 conn(0x7fd8340b3720 msgr2=0x7fd8340b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:32.870 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.865+0000 7fd83a742700 1 -- 172.21.15.91:0/1708067989 shutdown_connections 2023-11-24T06:08:32.870 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.865+0000 7fd83a742700 1 -- 172.21.15.91:0/1708067989 wait complete. 2023-11-24T06:08:32.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.866+0000 7fd83a742700 1 Processor -- start 2023-11-24T06:08:32.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.866+0000 7fd83a742700 1 -- start start 2023-11-24T06:08:32.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd83a742700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd83406fa80 0x7fd8340cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd83a742700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd8340b0e90 0x7fd8341f4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd83a742700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd83415e1c0 0x7fd83415f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd83a742700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8341f49d0 con 0x7fd8340b0e90 2023-11-24T06:08:32.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd83a742700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83415ee70 con 0x7fd83415e1c0 2023-11-24T06:08:32.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd83a742700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83415f840 con 0x7fd83406fa80 2023-11-24T06:08:32.874 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd8337fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd8340b0e90 0x7fd8341f4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.874 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd8337fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd8340b0e90 0x7fd8341f4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58780/0 (socket says 172.21.15.91:58780) 2023-11-24T06:08:32.874 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd8337fe700 1 -- 172.21.15.91:0/4064223412 learned_addr learned my addr 172.21.15.91:0/4064223412 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:32.874 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd833fff700 1 --2- 172.21.15.91:0/4064223412 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd83406fa80 0x7fd8340cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.874 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd838cdf700 1 --2- 172.21.15.91:0/4064223412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd83415e1c0 0x7fd83415f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.874 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd8337fe700 1 -- 172.21.15.91:0/4064223412 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd83406fa80 msgr2=0x7fd8340cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.874 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd8337fe700 1 --2- 172.21.15.91:0/4064223412 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd83406fa80 0x7fd8340cced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.874 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd8337fe700 1 -- 172.21.15.91:0/4064223412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd83415e1c0 msgr2=0x7fd83415f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:32.874 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd8337fe700 1 --2- 172.21.15.91:0/4064223412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd83415e1c0 0x7fd83415f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:32.874 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd8337fe700 1 -- 172.21.15.91:0/4064223412 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd824009550 con 0x7fd8340b0e90 2023-11-24T06:08:32.874 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd838cdf700 1 --2- 172.21.15.91:0/4064223412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd83415e1c0 0x7fd83415f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:32.875 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd8337fe700 1 --2- 172.21.15.91:0/4064223412 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd8340b0e90 0x7fd8341f4490 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd824009c40 tx=0x7fd824004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:32.875 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd830ff9700 1 -- 172.21.15.91:0/4064223412 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824004590 con 0x7fd8340b0e90 2023-11-24T06:08:32.875 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd830ff9700 1 -- 172.21.15.91:0/4064223412 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82401f070 con 0x7fd8340b0e90 2023-11-24T06:08:32.875 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd830ff9700 1 -- 172.21.15.91:0/4064223412 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8240038a0 con 0x7fd8340b0e90 2023-11-24T06:08:32.875 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.867+0000 7fd83a742700 1 -- 172.21.15.91:0/4064223412 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd83415e840 con 0x7fd8340b0e90 2023-11-24T06:08:32.875 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.868+0000 7fd83a742700 1 -- 172.21.15.91:0/4064223412 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8340cd8a0 con 0x7fd8340b0e90 2023-11-24T06:08:32.875 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.868+0000 7fd83a742700 1 -- 172.21.15.91:0/4064223412 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8340cda30 con 0x7fd8340b0e90 2023-11-24T06:08:32.875 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.869+0000 7fd830ff9700 1 -- 172.21.15.91:0/4064223412 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd824003a00 con 0x7fd8340b0e90 2023-11-24T06:08:32.875 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.869+0000 7fd830ff9700 1 --2- 172.21.15.91:0/4064223412 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd82006af00 0x7fd82006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:32.875 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.869+0000 7fd830ff9700 1 -- 172.21.15.91:0/4064223412 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd824021070 con 0x7fd8340b0e90 2023-11-24T06:08:32.875 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.869+0000 7fd833fff700 1 --2- 172.21.15.91:0/4064223412 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd82006af00 0x7fd82006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:32.875 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.870+0000 7fd833fff700 1 --2- 172.21.15.91:0/4064223412 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd82006af00 0x7fd82006d3c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd828006fd0 tx=0x7fd828008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:32.876 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:32.874+0000 7fd830ff9700 1 -- 172.21.15.91:0/4064223412 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd824055be0 con 0x7fd8340b0e90 2023-11-24T06:08:33.068 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.066+0000 7fd83a742700 1 -- 172.21.15.91:0/4064223412 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd834068670 con 0x7fd8340b0e90 2023-11-24T06:08:33.069 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.067+0000 7fd830ff9700 1 -- 172.21.15.91:0/4064223412 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7fd824015070 con 0x7fd8340b0e90 2023-11-24T06:08:33.070 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:33.070 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":20,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:02.415612+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:33.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.069+0000 7fd83a742700 1 -- 172.21.15.91:0/4064223412 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd82006af00 msgr2=0x7fd82006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.069+0000 7fd83a742700 1 --2- 172.21.15.91:0/4064223412 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd82006af00 0x7fd82006d3c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd828006fd0 tx=0x7fd828008040 comp rx=0 tx=0).stop 2023-11-24T06:08:33.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.069+0000 7fd83a742700 1 -- 172.21.15.91:0/4064223412 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd8340b0e90 msgr2=0x7fd8341f4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.069+0000 7fd83a742700 1 --2- 172.21.15.91:0/4064223412 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd8340b0e90 0x7fd8341f4490 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd824009c40 tx=0x7fd824004750 comp rx=0 tx=0).stop 2023-11-24T06:08:33.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.069+0000 7fd83a742700 1 -- 172.21.15.91:0/4064223412 shutdown_connections 2023-11-24T06:08:33.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.069+0000 7fd83a742700 1 --2- 172.21.15.91:0/4064223412 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd82006af00 0x7fd82006d3c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.069+0000 7fd83a742700 1 --2- 172.21.15.91:0/4064223412 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd83415e1c0 0x7fd83415f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.073 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.069+0000 7fd83a742700 1 --2- 172.21.15.91:0/4064223412 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd83406fa80 0x7fd8340cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.073 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.069+0000 7fd83a742700 1 --2- 172.21.15.91:0/4064223412 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd8340b0e90 0x7fd8341f4490 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.073 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.069+0000 7fd83a742700 1 -- 172.21.15.91:0/4064223412 >> 172.21.15.91:0/4064223412 conn(0x7fd8340b3720 msgr2=0x7fd8340b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:33.073 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.069+0000 7fd83a742700 1 -- 172.21.15.91:0/4064223412 shutdown_connections 2023-11-24T06:08:33.073 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.069+0000 7fd83a742700 1 -- 172.21.15.91:0/4064223412 wait complete. 2023-11-24T06:08:33.084 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:07:58.710106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:07:59.774819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:08:33.084 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:08:33.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.204+0000 7f730450b700 1 Processor -- start 2023-11-24T06:08:33.208 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.204+0000 7f730450b700 1 -- start start 2023-11-24T06:08:33.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.205+0000 7f730450b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72fc06b620 0x7f72fc0b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.205+0000 7f730450b700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f72fc0b5ab0 0x7f72fc0b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.205+0000 7f730450b700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f72fc1478c0 0x7f72fc161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.205+0000 7f730450b700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72fc071f50 con 0x7f72fc1478c0 2023-11-24T06:08:33.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.205+0000 7f730450b700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72fc072230 con 0x7f72fc0b5ab0 2023-11-24T06:08:33.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.205+0000 7f730450b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc072510 con 0x7f72fc06b620 2023-11-24T06:08:33.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.205+0000 7f7302aa8700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f72fc1478c0 0x7f72fc161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.205+0000 7f7302aa8700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f72fc1478c0 0x7f72fc161ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58790/0 (socket says 172.21.15.91:58790) 2023-11-24T06:08:33.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.205+0000 7f7302aa8700 1 -- 172.21.15.91:0/4090998857 learned_addr learned my addr 172.21.15.91:0/4090998857 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:33.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.205+0000 7f73022a7700 1 --2- 172.21.15.91:0/4090998857 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72fc06b620 0x7f72fc0b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.206+0000 7f7301aa6700 1 --2- 172.21.15.91:0/4090998857 >> v2:172.21.15.94:3300/0 conn(0x7f72fc0b5ab0 0x7f72fc0b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.206+0000 7f7302aa8700 1 -- 172.21.15.91:0/4090998857 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72fc06b620 msgr2=0x7f72fc0b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:33.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.206+0000 7f7302aa8700 1 --2- 172.21.15.91:0/4090998857 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72fc06b620 0x7f72fc0b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.206+0000 7f7302aa8700 1 -- 172.21.15.91:0/4090998857 >> v2:172.21.15.94:3300/0 conn(0x7f72fc0b5ab0 msgr2=0x7f72fc0b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:33.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.206+0000 7f7302aa8700 1 --2- 172.21.15.91:0/4090998857 >> v2:172.21.15.94:3300/0 conn(0x7f72fc0b5ab0 0x7f72fc0b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.206+0000 7f7302aa8700 1 -- 172.21.15.91:0/4090998857 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc0727f0 con 0x7f72fc1478c0 2023-11-24T06:08:33.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.206+0000 7f7302aa8700 1 --2- 172.21.15.91:0/4090998857 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72fc1478c0 0x7f72fc161ba0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f72fc069550 tx=0x7f72f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0918f698679334d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:33.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.206+0000 7f73012a5700 1 -- 172.21.15.91:0/4090998857 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f400e070 con 0x7f72fc1478c0 2023-11-24T06:08:33.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.206+0000 7f73012a5700 1 -- 172.21.15.91:0/4090998857 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f400b910 con 0x7f72fc1478c0 2023-11-24T06:08:33.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.206+0000 7f73012a5700 1 -- 172.21.15.91:0/4090998857 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f400bcc0 con 0x7f72fc1478c0 2023-11-24T06:08:33.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.206+0000 7f730450b700 1 -- 172.21.15.91:0/4090998857 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72fc1478c0 msgr2=0x7f72fc161ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.206+0000 7f730450b700 1 --2- 172.21.15.91:0/4090998857 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72fc1478c0 0x7f72fc161ba0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f72fc069550 tx=0x7f72f400b330 comp rx=0 tx=0).stop 2023-11-24T06:08:33.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.207+0000 7f730450b700 1 -- 172.21.15.91:0/4090998857 shutdown_connections 2023-11-24T06:08:33.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.207+0000 7f730450b700 1 --2- 172.21.15.91:0/4090998857 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72fc1478c0 0x7f72fc161ba0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.207+0000 7f730450b700 1 --2- 172.21.15.91:0/4090998857 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72fc06b620 0x7f72fc0b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.207+0000 7f730450b700 1 --2- 172.21.15.91:0/4090998857 >> v2:172.21.15.94:3300/0 conn(0x7f72fc0b5ab0 0x7f72fc0b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.207+0000 7f730450b700 1 -- 172.21.15.91:0/4090998857 >> 172.21.15.91:0/4090998857 conn(0x7f72fc0b2740 msgr2=0x7f72fc0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:33.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.207+0000 7f730450b700 1 -- 172.21.15.91:0/4090998857 shutdown_connections 2023-11-24T06:08:33.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.207+0000 7f730450b700 1 -- 172.21.15.91:0/4090998857 wait complete. 2023-11-24T06:08:33.215 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.208+0000 7f730450b700 1 Processor -- start 2023-11-24T06:08:33.215 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.208+0000 7f730450b700 1 -- start start 2023-11-24T06:08:33.215 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.209+0000 7f730450b700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72fc06b620 0x7f72fc145510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.215 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.209+0000 7f730450b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72fc0b5ab0 0x7f72fc0cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.215 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.209+0000 7f730450b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72fc1478c0 0x7f72fc0adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.215 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.209+0000 7f730450b700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc145ef0 con 0x7f72fc1478c0 2023-11-24T06:08:33.215 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.209+0000 7f730450b700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc0cc110 con 0x7f72fc06b620 2023-11-24T06:08:33.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.209+0000 7f730450b700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc0cd530 con 0x7f72fc0b5ab0 2023-11-24T06:08:33.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.209+0000 7f7302aa8700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72fc1478c0 0x7f72fc0adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.209+0000 7f7302aa8700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72fc1478c0 0x7f72fc0adad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58796/0 (socket says 172.21.15.91:58796) 2023-11-24T06:08:33.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.209+0000 7f7302aa8700 1 -- 172.21.15.91:0/758732840 learned_addr learned my addr 172.21.15.91:0/758732840 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:33.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.209+0000 7f7301aa6700 1 --2- 172.21.15.91:0/758732840 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72fc0b5ab0 0x7f72fc0cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.209+0000 7f73022a7700 1 --2- 172.21.15.91:0/758732840 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72fc06b620 0x7f72fc145510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.209+0000 7f7302aa8700 1 -- 172.21.15.91:0/758732840 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72fc0b5ab0 msgr2=0x7f72fc0cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.209+0000 7f7302aa8700 1 --2- 172.21.15.91:0/758732840 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72fc0b5ab0 0x7f72fc0cbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.210+0000 7f7302aa8700 1 -- 172.21.15.91:0/758732840 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72fc06b620 msgr2=0x7f72fc145510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.210+0000 7f7302aa8700 1 --2- 172.21.15.91:0/758732840 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72fc06b620 0x7f72fc145510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.210+0000 7f7302aa8700 1 -- 172.21.15.91:0/758732840 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f4009b30 con 0x7f72fc1478c0 2023-11-24T06:08:33.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.210+0000 7f73022a7700 1 --2- 172.21.15.91:0/758732840 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72fc06b620 0x7f72fc145510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:33.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.210+0000 7f7302aa8700 1 --2- 172.21.15.91:0/758732840 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72fc1478c0 0x7f72fc0adad0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0d1bd0 tx=0x7f72f4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:33.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.210+0000 7f72f2ffd700 1 -- 172.21.15.91:0/758732840 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f40099b0 con 0x7f72fc1478c0 2023-11-24T06:08:33.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.210+0000 7f72f2ffd700 1 -- 172.21.15.91:0/758732840 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f40092e0 con 0x7f72fc1478c0 2023-11-24T06:08:33.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.210+0000 7f730450b700 1 -- 172.21.15.91:0/758732840 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72fc145c10 con 0x7f72fc1478c0 2023-11-24T06:08:33.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.210+0000 7f72f2ffd700 1 -- 172.21.15.91:0/758732840 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f4004470 con 0x7f72fc1478c0 2023-11-24T06:08:33.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.210+0000 7f730450b700 1 -- 172.21.15.91:0/758732840 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72fc0cdab0 con 0x7f72fc1478c0 2023-11-24T06:08:33.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.211+0000 7f730450b700 1 -- 172.21.15.91:0/758732840 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72fc0ce100 con 0x7f72fc1478c0 2023-11-24T06:08:33.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.211+0000 7f72f2ffd700 1 -- 172.21.15.91:0/758732840 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f72f40045d0 con 0x7f72fc1478c0 2023-11-24T06:08:33.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.212+0000 7f72f2ffd700 1 --2- 172.21.15.91:0/758732840 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f72e006aed0 0x7f72e006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.212+0000 7f73022a7700 1 --2- 172.21.15.91:0/758732840 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f72e006aed0 0x7f72e006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.212+0000 7f72f2ffd700 1 -- 172.21.15.91:0/758732840 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72f408a6f0 con 0x7f72fc1478c0 2023-11-24T06:08:33.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.212+0000 7f73022a7700 1 --2- 172.21.15.91:0/758732840 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f72e006aed0 0x7f72e006d390 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f72ec009bb0 tx=0x7f72ec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:33.219 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.216+0000 7f72f2ffd700 1 -- 172.21.15.91:0/758732840 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f72f4059240 con 0x7f72fc1478c0 2023-11-24T06:08:33.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.405+0000 7f730450b700 1 -- 172.21.15.91:0/758732840 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f72fc075480 con 0x7f72fc1478c0 2023-11-24T06:08:33.409 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.406+0000 7f72f2ffd700 1 -- 172.21.15.91:0/758732840 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f72f401f020 con 0x7f72fc1478c0 2023-11-24T06:08:33.409 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:08:33.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.408+0000 7f730450b700 1 -- 172.21.15.91:0/758732840 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f72e006aed0 msgr2=0x7f72e006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.408+0000 7f730450b700 1 --2- 172.21.15.91:0/758732840 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f72e006aed0 0x7f72e006d390 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f72ec009bb0 tx=0x7f72ec008040 comp rx=0 tx=0).stop 2023-11-24T06:08:33.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.408+0000 7f730450b700 1 -- 172.21.15.91:0/758732840 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72fc1478c0 msgr2=0x7f72fc0adad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.408+0000 7f730450b700 1 --2- 172.21.15.91:0/758732840 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72fc1478c0 0x7f72fc0adad0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0d1bd0 tx=0x7f72f4004ad0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.408+0000 7f730450b700 1 -- 172.21.15.91:0/758732840 shutdown_connections 2023-11-24T06:08:33.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.408+0000 7f730450b700 1 --2- 172.21.15.91:0/758732840 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f72e006aed0 0x7f72e006d390 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.408+0000 7f730450b700 1 --2- 172.21.15.91:0/758732840 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72fc06b620 0x7f72fc145510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.408+0000 7f730450b700 1 --2- 172.21.15.91:0/758732840 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72fc1478c0 0x7f72fc0adad0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.408+0000 7f730450b700 1 --2- 172.21.15.91:0/758732840 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72fc0b5ab0 0x7f72fc0cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.408+0000 7f730450b700 1 -- 172.21.15.91:0/758732840 >> 172.21.15.91:0/758732840 conn(0x7f72fc0b2740 msgr2=0x7f72fc0cfde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:33.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.408+0000 7f730450b700 1 -- 172.21.15.91:0/758732840 shutdown_connections 2023-11-24T06:08:33.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.408+0000 7f730450b700 1 -- 172.21.15.91:0/758732840 wait complete. 2023-11-24T06:08:33.423 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:08:33.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.544+0000 7f9460fcb700 1 Processor -- start 2023-11-24T06:08:33.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.544+0000 7f9460fcb700 1 -- start start 2023-11-24T06:08:33.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.545+0000 7f9460fcb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f945c143ae0 0x7f945c0b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.545+0000 7f9460fcb700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f945c0722c0 0x7f945c06e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.545+0000 7f9460fcb700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945c143ec0 con 0x7f945c1440f0 2023-11-24T06:08:33.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.545+0000 7f9460fcb700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c0b40a0 con 0x7f945c0722c0 2023-11-24T06:08:33.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.545+0000 7f9460fcb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c0b3ae0 con 0x7f945c143ae0 2023-11-24T06:08:33.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.545+0000 7f945ad9d700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f945c0722c0 0x7f945c06e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.545+0000 7f945a59c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f945c143ae0 0x7f945c0b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.545+0000 7f945a59c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f945c143ae0 0x7f945c0b51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:42232/0 (socket says 172.21.15.91:42232) 2023-11-24T06:08:33.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.545+0000 7f945a59c700 1 -- 172.21.15.91:0/3992897048 learned_addr learned my addr 172.21.15.91:0/3992897048 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:33.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.545+0000 7f945a59c700 1 -- 172.21.15.91:0/3992897048 >> v2:172.21.15.94:3300/0 conn(0x7f945c0722c0 msgr2=0x7f945c06e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:33.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.545+0000 7f945a59c700 1 --2- 172.21.15.91:0/3992897048 >> v2:172.21.15.94:3300/0 conn(0x7f945c0722c0 0x7f945c06e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.545+0000 7f945a59c700 1 -- 172.21.15.91:0/3992897048 >> v1:172.21.15.94:6789/0 conn(0x7f945c1440f0 legacy=0x7f945c071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.545+0000 7f945a59c700 1 -- 172.21.15.91:0/3992897048 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c0b3dc0 con 0x7f945c143ae0 2023-11-24T06:08:33.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.546+0000 7f945a59c700 1 --2- 172.21.15.91:0/3992897048 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f945c143ae0 0x7f945c0b51b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f944c0098f0 tx=0x7f944c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6b03a735cdc7669 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:33.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.546+0000 7f945959a700 1 -- 172.21.15.91:0/3992897048 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c011830 con 0x7f945c143ae0 2023-11-24T06:08:33.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.546+0000 7f945959a700 1 -- 172.21.15.91:0/3992897048 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944c011990 con 0x7f945c143ae0 2023-11-24T06:08:33.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.546+0000 7f945959a700 1 -- 172.21.15.91:0/3992897048 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c011d20 con 0x7f945c143ae0 2023-11-24T06:08:33.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.546+0000 7f9460fcb700 1 -- 172.21.15.91:0/3992897048 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f945c143ae0 msgr2=0x7f945c0b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.546+0000 7f9460fcb700 1 --2- 172.21.15.91:0/3992897048 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f945c143ae0 0x7f945c0b51b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f944c0098f0 tx=0x7f944c009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:33.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.546+0000 7f9460fcb700 1 -- 172.21.15.91:0/3992897048 shutdown_connections 2023-11-24T06:08:33.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.546+0000 7f9460fcb700 1 --2- 172.21.15.91:0/3992897048 >> v2:172.21.15.94:3300/0 conn(0x7f945c0722c0 0x7f945c06e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.546+0000 7f9460fcb700 1 --2- 172.21.15.91:0/3992897048 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f945c143ae0 0x7f945c0b51b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.555 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.546+0000 7f9460fcb700 1 -- 172.21.15.91:0/3992897048 >> 172.21.15.91:0/3992897048 conn(0x7f945c0b2740 msgr2=0x7f945c0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:33.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.546+0000 7f9460fcb700 1 -- 172.21.15.91:0/3992897048 shutdown_connections 2023-11-24T06:08:33.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.546+0000 7f9460fcb700 1 -- 172.21.15.91:0/3992897048 wait complete. 2023-11-24T06:08:33.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.548+0000 7f9460fcb700 1 Processor -- start 2023-11-24T06:08:33.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.548+0000 7f9460fcb700 1 -- start start 2023-11-24T06:08:33.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.548+0000 7f9460fcb700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f945c0722c0 0x7f945c15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.548+0000 7f9460fcb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f945c143ae0 0x7f945c132410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.548+0000 7f9460fcb700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f945c1440f0 0x7f945c15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.548+0000 7f9460fcb700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c13fe10 con 0x7f945c0722c0 2023-11-24T06:08:33.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f9460fcb700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c15c240 con 0x7f945c1440f0 2023-11-24T06:08:33.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f9460fcb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c153bf0 con 0x7f945c143ae0 2023-11-24T06:08:33.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f945a59c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f945c0722c0 0x7f945c15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f945a59c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f945c0722c0 0x7f945c15d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58806/0 (socket says 172.21.15.91:58806) 2023-11-24T06:08:33.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f945a59c700 1 -- 172.21.15.91:0/3056332583 learned_addr learned my addr 172.21.15.91:0/3056332583 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:33.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f945ad9d700 1 --2- 172.21.15.91:0/3056332583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f945c1440f0 0x7f945c15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f9459d9b700 1 --2- 172.21.15.91:0/3056332583 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f945c143ae0 0x7f945c132410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f945a59c700 1 -- 172.21.15.91:0/3056332583 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f945c143ae0 msgr2=0x7f945c132410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f945a59c700 1 --2- 172.21.15.91:0/3056332583 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f945c143ae0 0x7f945c132410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f945a59c700 1 -- 172.21.15.91:0/3056332583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f945c1440f0 msgr2=0x7f945c15bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f945a59c700 1 --2- 172.21.15.91:0/3056332583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f945c1440f0 0x7f945c15bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f945a59c700 1 -- 172.21.15.91:0/3056332583 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944c009550 con 0x7f945c0722c0 2023-11-24T06:08:33.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f945ad9d700 1 --2- 172.21.15.91:0/3056332583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f945c1440f0 0x7f945c15bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:33.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f945a59c700 1 --2- 172.21.15.91:0/3056332583 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f945c0722c0 0x7f945c15d5f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f944c000c00 tx=0x7f944c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:33.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f944affd700 1 -- 172.21.15.91:0/3056332583 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c004480 con 0x7f945c0722c0 2023-11-24T06:08:33.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f944affd700 1 -- 172.21.15.91:0/3056332583 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944c01f070 con 0x7f945c0722c0 2023-11-24T06:08:33.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f9460fcb700 1 -- 172.21.15.91:0/3056332583 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945c153ed0 con 0x7f945c0722c0 2023-11-24T06:08:33.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f944affd700 1 -- 172.21.15.91:0/3056332583 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944c0038a0 con 0x7f945c0722c0 2023-11-24T06:08:33.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.549+0000 7f9460fcb700 1 -- 172.21.15.91:0/3056332583 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945c154490 con 0x7f945c0722c0 2023-11-24T06:08:33.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.550+0000 7f9460fcb700 1 -- 172.21.15.91:0/3056332583 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945c075480 con 0x7f945c0722c0 2023-11-24T06:08:33.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.551+0000 7f944affd700 1 -- 172.21.15.91:0/3056332583 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f944c021030 con 0x7f945c0722c0 2023-11-24T06:08:33.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.551+0000 7f944affd700 1 --2- 172.21.15.91:0/3056332583 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f943c06aed0 0x7f943c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.551+0000 7f944affd700 1 -- 172.21.15.91:0/3056332583 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f944c08ab10 con 0x7f945c0722c0 2023-11-24T06:08:33.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.551+0000 7f9459d9b700 1 --2- 172.21.15.91:0/3056332583 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f943c06aed0 0x7f943c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.558 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.551+0000 7f9459d9b700 1 --2- 172.21.15.91:0/3056332583 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f943c06aed0 0x7f943c06d390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f945c13ec30 tx=0x7f9450009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:33.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.555+0000 7f944affd700 1 -- 172.21.15.91:0/3056332583 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f944c055ce0 con 0x7f945c0722c0 2023-11-24T06:08:33.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.747+0000 7f9460fcb700 1 -- 172.21.15.91:0/3056332583 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f945c154ae0 con 0x7f945c0722c0 2023-11-24T06:08:33.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.747+0000 7f944affd700 1 -- 172.21.15.91:0/3056332583 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f944c059300 con 0x7f945c0722c0 2023-11-24T06:08:33.751 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 32 2023-11-24T06:08:33.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.749+0000 7f9460fcb700 1 -- 172.21.15.91:0/3056332583 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f943c06aed0 msgr2=0x7f943c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.749+0000 7f9460fcb700 1 --2- 172.21.15.91:0/3056332583 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f943c06aed0 0x7f943c06d390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f945c13ec30 tx=0x7f9450009040 comp rx=0 tx=0).stop 2023-11-24T06:08:33.753 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.749+0000 7f9460fcb700 1 -- 172.21.15.91:0/3056332583 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f945c0722c0 msgr2=0x7f945c15d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.753 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.749+0000 7f9460fcb700 1 --2- 172.21.15.91:0/3056332583 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f945c0722c0 0x7f945c15d5f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f944c000c00 tx=0x7f944c004750 comp rx=0 tx=0).stop 2023-11-24T06:08:33.753 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.749+0000 7f9460fcb700 1 -- 172.21.15.91:0/3056332583 shutdown_connections 2023-11-24T06:08:33.753 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.749+0000 7f9460fcb700 1 --2- 172.21.15.91:0/3056332583 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f943c06aed0 0x7f943c06d390 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.753 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.749+0000 7f9460fcb700 1 --2- 172.21.15.91:0/3056332583 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f945c1440f0 0x7f945c15bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.754 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.749+0000 7f9460fcb700 1 --2- 172.21.15.91:0/3056332583 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f945c0722c0 0x7f945c15d5f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.754 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.749+0000 7f9460fcb700 1 --2- 172.21.15.91:0/3056332583 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f945c143ae0 0x7f945c132410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.754 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.749+0000 7f9460fcb700 1 -- 172.21.15.91:0/3056332583 >> 172.21.15.91:0/3056332583 conn(0x7f945c0b2740 msgr2=0x7f945c0cf540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:33.754 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.749+0000 7f9460fcb700 1 -- 172.21.15.91:0/3056332583 shutdown_connections 2023-11-24T06:08:33.754 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.749+0000 7f9460fcb700 1 -- 172.21.15.91:0/3056332583 wait complete. 2023-11-24T06:08:33.764 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:08:33.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.885+0000 7f197fadb700 1 Processor -- start 2023-11-24T06:08:33.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.885+0000 7f197fadb700 1 -- start start 2023-11-24T06:08:33.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.886+0000 7f197fadb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f197815e1c0 0x7f197806eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.886+0000 7f197fadb700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f197806fc50 0x7f197806ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.886+0000 7f197fadb700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19780d1230 con 0x7f19780b0e90 2023-11-24T06:08:33.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.886+0000 7f197fadb700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19780d0c70 con 0x7f197806fc50 2023-11-24T06:08:33.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.886+0000 7f197fadb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19780d0f50 con 0x7f197815e1c0 2023-11-24T06:08:33.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.886+0000 7f197e078700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f197806fc50 0x7f197806ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.886+0000 7f197d877700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f197815e1c0 0x7f197806eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.886+0000 7f197e078700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f197806fc50 0x7f197806ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:43210/0 (socket says 172.21.15.91:43210) 2023-11-24T06:08:33.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.886+0000 7f197e078700 1 -- 172.21.15.91:0/713505614 learned_addr learned my addr 172.21.15.91:0/713505614 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:33.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.887+0000 7f197e078700 1 -- 172.21.15.91:0/713505614 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f197815e1c0 msgr2=0x7f197806eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:33.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.887+0000 7f197e078700 1 --2- 172.21.15.91:0/713505614 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f197815e1c0 0x7f197806eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.887+0000 7f197e078700 1 -- 172.21.15.91:0/713505614 >> v1:172.21.15.94:6789/0 conn(0x7f19780b0e90 legacy=0x7f197806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.887+0000 7f197e078700 1 -- 172.21.15.91:0/713505614 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197806c210 con 0x7f197806fc50 2023-11-24T06:08:33.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.887+0000 7f197e078700 1 --2- 172.21.15.91:0/713505614 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f197806fc50 0x7f197806ca10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1970009e50 tx=0x7f197000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d94719e2b0d013fb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:33.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.887+0000 7f197c875700 1 -- 172.21.15.91:0/713505614 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197000e070 con 0x7f197806fc50 2023-11-24T06:08:33.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.887+0000 7f197c875700 1 -- 172.21.15.91:0/713505614 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197000b910 con 0x7f197806fc50 2023-11-24T06:08:33.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.887+0000 7f197c875700 1 -- 172.21.15.91:0/713505614 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197000bcc0 con 0x7f197806fc50 2023-11-24T06:08:33.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.887+0000 7f197fadb700 1 -- 172.21.15.91:0/713505614 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f197806fc50 msgr2=0x7f197806ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.887+0000 7f197fadb700 1 --2- 172.21.15.91:0/713505614 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f197806fc50 0x7f197806ca10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1970009e50 tx=0x7f197000b330 comp rx=0 tx=0).stop 2023-11-24T06:08:33.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.887+0000 7f197fadb700 1 -- 172.21.15.91:0/713505614 shutdown_connections 2023-11-24T06:08:33.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.887+0000 7f197fadb700 1 --2- 172.21.15.91:0/713505614 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f197806fc50 0x7f197806ca10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.887+0000 7f197fadb700 1 --2- 172.21.15.91:0/713505614 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f197815e1c0 0x7f197806eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.887+0000 7f197fadb700 1 -- 172.21.15.91:0/713505614 >> 172.21.15.91:0/713505614 conn(0x7f19780b3720 msgr2=0x7f19780b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:33.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.888+0000 7f197fadb700 1 -- 172.21.15.91:0/713505614 shutdown_connections 2023-11-24T06:08:33.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.888+0000 7f197fadb700 1 -- 172.21.15.91:0/713505614 wait complete. 2023-11-24T06:08:33.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.889+0000 7f197fadb700 1 Processor -- start 2023-11-24T06:08:33.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.889+0000 7f197fadb700 1 -- start start 2023-11-24T06:08:33.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.889+0000 7f197fadb700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f197806fc50 0x7f19781532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.889+0000 7f197fadb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f19780b0e90 0x7f19780b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.889+0000 7f197fadb700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f197815e1c0 0x7f19780b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.889+0000 7f197fadb700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19781f7600 con 0x7f197806fc50 2023-11-24T06:08:33.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.889+0000 7f197fadb700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19781f78e0 con 0x7f197815e1c0 2023-11-24T06:08:33.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.889+0000 7f197fadb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19781f7bc0 con 0x7f19780b0e90 2023-11-24T06:08:33.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f197d877700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f197806fc50 0x7f19781532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f197d877700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f197806fc50 0x7f19781532a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58820/0 (socket says 172.21.15.91:58820) 2023-11-24T06:08:33.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f197d877700 1 -- 172.21.15.91:0/3266711760 learned_addr learned my addr 172.21.15.91:0/3266711760 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:33.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f197e078700 1 --2- 172.21.15.91:0/3266711760 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f197815e1c0 0x7f19780b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f197d076700 1 --2- 172.21.15.91:0/3266711760 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f19780b0e90 0x7f19780b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f197d877700 1 -- 172.21.15.91:0/3266711760 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f19780b0e90 msgr2=0x7f19780b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f197d877700 1 --2- 172.21.15.91:0/3266711760 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f19780b0e90 0x7f19780b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f197d877700 1 -- 172.21.15.91:0/3266711760 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f197815e1c0 msgr2=0x7f19780b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:33.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f197d877700 1 --2- 172.21.15.91:0/3266711760 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f197815e1c0 0x7f19780b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:33.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f197d877700 1 -- 172.21.15.91:0/3266711760 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1970009b30 con 0x7f197806fc50 2023-11-24T06:08:33.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f197d076700 1 --2- 172.21.15.91:0/3266711760 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f19780b0e90 0x7f19780b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:33.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f197d877700 1 --2- 172.21.15.91:0/3266711760 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f197806fc50 0x7f19781532a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f197400e870 tx=0x7f197400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:33.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f196e7fc700 1 -- 172.21.15.91:0/3266711760 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1974009590 con 0x7f197806fc50 2023-11-24T06:08:33.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f196e7fc700 1 -- 172.21.15.91:0/3266711760 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1974013680 con 0x7f197806fc50 2023-11-24T06:08:33.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f196e7fc700 1 -- 172.21.15.91:0/3266711760 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19740139f0 con 0x7f197806fc50 2023-11-24T06:08:33.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f197fadb700 1 -- 172.21.15.91:0/3266711760 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19781f7ea0 con 0x7f197806fc50 2023-11-24T06:08:33.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.890+0000 7f197fadb700 1 -- 172.21.15.91:0/3266711760 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19781f83c0 con 0x7f197806fc50 2023-11-24T06:08:33.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.891+0000 7f197fadb700 1 -- 172.21.15.91:0/3266711760 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f197806ca10 con 0x7f197806fc50 2023-11-24T06:08:33.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.892+0000 7f196e7fc700 1 -- 172.21.15.91:0/3266711760 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1974013b50 con 0x7f197806fc50 2023-11-24T06:08:33.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.892+0000 7f196e7fc700 1 --2- 172.21.15.91:0/3266711760 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f195c06aed0 0x7f195c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:33.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.892+0000 7f197d076700 1 --2- 172.21.15.91:0/3266711760 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f195c06aed0 0x7f195c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:33.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.892+0000 7f196e7fc700 1 -- 172.21.15.91:0/3266711760 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1974089550 con 0x7f197806fc50 2023-11-24T06:08:33.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.892+0000 7f197d076700 1 --2- 172.21.15.91:0/3266711760 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f195c06aed0 0x7f195c06d390 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f19780c8670 tx=0x7f1968009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:33.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:33.896+0000 7f196e7fc700 1 -- 172.21.15.91:0/3266711760 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1974054580 con 0x7f197806fc50 2023-11-24T06:08:34.090 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.087+0000 7f197fadb700 1 -- 172.21.15.91:0/3266711760 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f19781f7740 con 0x7f197806fc50 2023-11-24T06:08:34.090 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.087+0000 7f196e7fc700 1 -- 172.21.15.91:0/3266711760 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f19781f7740 con 0x7f197806fc50 2023-11-24T06:08:34.090 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 64 2023-11-24T06:08:34.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.089+0000 7f197fadb700 1 -- 172.21.15.91:0/3266711760 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f195c06aed0 msgr2=0x7f195c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:34.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.089+0000 7f197fadb700 1 --2- 172.21.15.91:0/3266711760 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f195c06aed0 0x7f195c06d390 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f19780c8670 tx=0x7f1968009040 comp rx=0 tx=0).stop 2023-11-24T06:08:34.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.089+0000 7f197fadb700 1 -- 172.21.15.91:0/3266711760 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f197806fc50 msgr2=0x7f19781532a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:34.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.089+0000 7f197fadb700 1 --2- 172.21.15.91:0/3266711760 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f197806fc50 0x7f19781532a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f197400e870 tx=0x7f197400eb80 comp rx=0 tx=0).stop 2023-11-24T06:08:34.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.090+0000 7f197fadb700 1 -- 172.21.15.91:0/3266711760 shutdown_connections 2023-11-24T06:08:34.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.090+0000 7f197fadb700 1 --2- 172.21.15.91:0/3266711760 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f195c06aed0 0x7f195c06d390 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.090+0000 7f197fadb700 1 --2- 172.21.15.91:0/3266711760 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f197815e1c0 0x7f19780b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.090+0000 7f197fadb700 1 --2- 172.21.15.91:0/3266711760 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f197806fc50 0x7f19781532a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.090+0000 7f197fadb700 1 --2- 172.21.15.91:0/3266711760 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f19780b0e90 0x7f19780b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.090+0000 7f197fadb700 1 -- 172.21.15.91:0/3266711760 >> 172.21.15.91:0/3266711760 conn(0x7f19780b3720 msgr2=0x7f19780aea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:34.093 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.090+0000 7f197fadb700 1 -- 172.21.15.91:0/3266711760 shutdown_connections 2023-11-24T06:08:34.094 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.090+0000 7f197fadb700 1 -- 172.21.15.91:0/3266711760 wait complete. 2023-11-24T06:08:34.104 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_failed' 2023-11-24T06:08:34.225 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.222+0000 7f406f424700 1 Processor -- start 2023-11-24T06:08:34.225 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.222+0000 7f406f424700 1 -- start start 2023-11-24T06:08:34.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406f424700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f406806b620 0x7f40680b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:34.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406f424700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40680b5ab0 0x7f40680b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:34.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406f424700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f40681478c0 0x7f4068161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:34.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406f424700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4068071f50 con 0x7f406806b620 2023-11-24T06:08:34.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406f424700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4068072230 con 0x7f40681478c0 2023-11-24T06:08:34.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406f424700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4068072510 con 0x7f40680b5ab0 2023-11-24T06:08:34.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406d1c0700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f406806b620 0x7f40680b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:34.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406d1c0700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f406806b620 0x7f40680b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58828/0 (socket says 172.21.15.91:58828) 2023-11-24T06:08:34.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406d1c0700 1 -- 172.21.15.91:0/4245088669 learned_addr learned my addr 172.21.15.91:0/4245088669 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:34.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406c9bf700 1 --2- 172.21.15.91:0/4245088669 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40680b5ab0 0x7f40680b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:34.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406d9c1700 1 --2- 172.21.15.91:0/4245088669 >> v2:172.21.15.94:3300/0 conn(0x7f40681478c0 0x7f4068161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:34.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406d1c0700 1 -- 172.21.15.91:0/4245088669 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40680b5ab0 msgr2=0x7f40680b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:34.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406d1c0700 1 --2- 172.21.15.91:0/4245088669 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40680b5ab0 0x7f40680b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406d1c0700 1 -- 172.21.15.91:0/4245088669 >> v2:172.21.15.94:3300/0 conn(0x7f40681478c0 msgr2=0x7f4068161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:34.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406d1c0700 1 --2- 172.21.15.91:0/4245088669 >> v2:172.21.15.94:3300/0 conn(0x7f40681478c0 0x7f4068161ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406d1c0700 1 -- 172.21.15.91:0/4245088669 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40680727f0 con 0x7f406806b620 2023-11-24T06:08:34.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406c9bf700 1 --2- 172.21.15.91:0/4245088669 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40680b5ab0 0x7f40680b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:34.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f406d1c0700 1 --2- 172.21.15.91:0/4245088669 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f406806b620 0x7f40680b3a40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f40680747c0 tx=0x7f4064009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bed269f2d518960 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:34.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.223+0000 7f405ffff700 1 -- 172.21.15.91:0/4245088669 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406400b870 con 0x7f406806b620 2023-11-24T06:08:34.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.224+0000 7f405ffff700 1 -- 172.21.15.91:0/4245088669 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406400b9d0 con 0x7f406806b620 2023-11-24T06:08:34.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.224+0000 7f405ffff700 1 -- 172.21.15.91:0/4245088669 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406400bd60 con 0x7f406806b620 2023-11-24T06:08:34.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.224+0000 7f406f424700 1 -- 172.21.15.91:0/4245088669 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f406806b620 msgr2=0x7f40680b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:34.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.224+0000 7f406f424700 1 --2- 172.21.15.91:0/4245088669 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f406806b620 0x7f40680b3a40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f40680747c0 tx=0x7f4064009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.224+0000 7f406f424700 1 -- 172.21.15.91:0/4245088669 shutdown_connections 2023-11-24T06:08:34.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.224+0000 7f406f424700 1 --2- 172.21.15.91:0/4245088669 >> v2:172.21.15.94:3300/0 conn(0x7f40681478c0 0x7f4068161ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.224+0000 7f406f424700 1 --2- 172.21.15.91:0/4245088669 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f406806b620 0x7f40680b3a40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.224+0000 7f406f424700 1 --2- 172.21.15.91:0/4245088669 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40680b5ab0 0x7f40680b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.224+0000 7f406f424700 1 -- 172.21.15.91:0/4245088669 >> 172.21.15.91:0/4245088669 conn(0x7f40680b2740 msgr2=0x7f40680b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:34.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.224+0000 7f406f424700 1 -- 172.21.15.91:0/4245088669 shutdown_connections 2023-11-24T06:08:34.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.224+0000 7f406f424700 1 -- 172.21.15.91:0/4245088669 wait complete. 2023-11-24T06:08:34.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.225+0000 7f406f424700 1 Processor -- start 2023-11-24T06:08:34.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.225+0000 7f406f424700 1 -- start start 2023-11-24T06:08:34.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.226+0000 7f406f424700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f406806b620 0x7f40680cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:34.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.226+0000 7f406f424700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40680b5ab0 0x7f40680c8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:34.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.226+0000 7f406f424700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40681478c0 0x7f40680cb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:34.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.226+0000 7f406f424700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40681457c0 con 0x7f406806b620 2023-11-24T06:08:34.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.226+0000 7f406f424700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40680c9060 con 0x7f40681478c0 2023-11-24T06:08:34.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.226+0000 7f406f424700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40680c6c20 con 0x7f40680b5ab0 2023-11-24T06:08:34.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.226+0000 7f406d1c0700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f406806b620 0x7f40680cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:34.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.226+0000 7f406d1c0700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f406806b620 0x7f40680cfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58832/0 (socket says 172.21.15.91:58832) 2023-11-24T06:08:34.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.226+0000 7f406d1c0700 1 -- 172.21.15.91:0/423038185 learned_addr learned my addr 172.21.15.91:0/423038185 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:34.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.226+0000 7f406d1c0700 1 -- 172.21.15.91:0/423038185 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40680b5ab0 msgr2=0x7f40680c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:34.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.226+0000 7f406c9bf700 1 --2- 172.21.15.91:0/423038185 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40680b5ab0 0x7f40680c8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:34.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.226+0000 7f406d9c1700 1 --2- 172.21.15.91:0/423038185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40681478c0 0x7f40680cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:34.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.226+0000 7f406d1c0700 1 --2- 172.21.15.91:0/423038185 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40680b5ab0 0x7f40680c8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.226+0000 7f406d1c0700 1 -- 172.21.15.91:0/423038185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40681478c0 msgr2=0x7f40680cb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:34.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.226+0000 7f406d1c0700 1 --2- 172.21.15.91:0/423038185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40681478c0 0x7f40680cb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.226+0000 7f406d1c0700 1 -- 172.21.15.91:0/423038185 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4064009550 con 0x7f406806b620 2023-11-24T06:08:34.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.227+0000 7f406d1c0700 1 --2- 172.21.15.91:0/423038185 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f406806b620 0x7f40680cfea0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f40680d1bd0 tx=0x7f4064004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:34.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.227+0000 7f405dffb700 1 -- 172.21.15.91:0/423038185 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4064004310 con 0x7f406806b620 2023-11-24T06:08:34.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.227+0000 7f405dffb700 1 -- 172.21.15.91:0/423038185 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406401f070 con 0x7f406806b620 2023-11-24T06:08:34.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.227+0000 7f406f424700 1 -- 172.21.15.91:0/423038185 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40680c6f00 con 0x7f406806b620 2023-11-24T06:08:34.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.227+0000 7f405dffb700 1 -- 172.21.15.91:0/423038185 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40640036a0 con 0x7f406806b620 2023-11-24T06:08:34.235 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.227+0000 7f406f424700 1 -- 172.21.15.91:0/423038185 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40680c74c0 con 0x7f406806b620 2023-11-24T06:08:34.235 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.228+0000 7f406f424700 1 -- 172.21.15.91:0/423038185 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4068068670 con 0x7f406806b620 2023-11-24T06:08:34.235 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.228+0000 7f405dffb700 1 -- 172.21.15.91:0/423038185 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4064021030 con 0x7f406806b620 2023-11-24T06:08:34.235 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.229+0000 7f405dffb700 1 --2- 172.21.15.91:0/423038185 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f404c06aed0 0x7f404c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:34.235 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.229+0000 7f405dffb700 1 -- 172.21.15.91:0/423038185 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f406408aa30 con 0x7f406806b620 2023-11-24T06:08:34.235 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.229+0000 7f406c9bf700 1 --2- 172.21.15.91:0/423038185 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f404c06aed0 0x7f404c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:34.235 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.229+0000 7f406c9bf700 1 --2- 172.21.15.91:0/423038185 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f404c06aed0 0x7f404c06d390 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f40681e5050 tx=0x7f4058008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:34.236 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.233+0000 7f405dffb700 1 -- 172.21.15.91:0/423038185 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4064055c90 con 0x7f406806b620 2023-11-24T06:08:34.505 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.502+0000 7f406f424700 1 -- 172.21.15.91:0/423038185 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_failed"]} v 0) v1 -- 0x7f40680d16b0 con 0x7f406806b620 2023-11-24T06:08:34.774 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.771+0000 7f405dffb700 1 -- 172.21.15.91:0/423038185 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_failed"]}]=0 v55) v1 ==== 168+0+0 (secure 0 0 0) 0x7f40640592b0 con 0x7f406806b620 2023-11-24T06:08:34.776 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.773+0000 7f406f424700 1 -- 172.21.15.91:0/423038185 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f404c06aed0 msgr2=0x7f404c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:34.776 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.773+0000 7f406f424700 1 --2- 172.21.15.91:0/423038185 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f404c06aed0 0x7f404c06d390 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f40681e5050 tx=0x7f4058008040 comp rx=0 tx=0).stop 2023-11-24T06:08:34.776 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.773+0000 7f406f424700 1 -- 172.21.15.91:0/423038185 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f406806b620 msgr2=0x7f40680cfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:34.776 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.773+0000 7f406f424700 1 --2- 172.21.15.91:0/423038185 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f406806b620 0x7f40680cfea0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f40680d1bd0 tx=0x7f4064004750 comp rx=0 tx=0).stop 2023-11-24T06:08:34.776 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.773+0000 7f406f424700 1 -- 172.21.15.91:0/423038185 shutdown_connections 2023-11-24T06:08:34.776 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.773+0000 7f406f424700 1 --2- 172.21.15.91:0/423038185 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f404c06aed0 0x7f404c06d390 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.773+0000 7f406f424700 1 --2- 172.21.15.91:0/423038185 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f40681478c0 0x7f40680cb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.773+0000 7f406f424700 1 --2- 172.21.15.91:0/423038185 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f406806b620 0x7f40680cfea0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.773+0000 7f406f424700 1 --2- 172.21.15.91:0/423038185 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f40680b5ab0 0x7f40680c8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.773+0000 7f406f424700 1 -- 172.21.15.91:0/423038185 >> 172.21.15.91:0/423038185 conn(0x7f40680b2740 msgr2=0x7f40680b0720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:34.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.773+0000 7f406f424700 1 -- 172.21.15.91:0/423038185 shutdown_connections 2023-11-24T06:08:34.777 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.773+0000 7f406f424700 1 -- 172.21.15.91:0/423038185 wait complete. 2023-11-24T06:08:34.788 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T06:08:34.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.909+0000 7f7b52ad4700 1 Processor -- start 2023-11-24T06:08:34.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.909+0000 7f7b52ad4700 1 -- start start 2023-11-24T06:08:34.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.910+0000 7f7b52ad4700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f7b4c15bde0 0x7f7b4c15c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:34.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.910+0000 7f7b52ad4700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7b4c0ce5c0 0x7f7b4c0cea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:34.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.910+0000 7f7b52ad4700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4c0d0a90 con 0x7f7b4c0cf2b0 2023-11-24T06:08:34.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.910+0000 7f7b52ad4700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b4c0cef70 con 0x7f7b4c15bde0 2023-11-24T06:08:34.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.910+0000 7f7b52ad4700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b4c06ee10 con 0x7f7b4c0ce5c0 2023-11-24T06:08:34.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.910+0000 7f7b50870700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f7b4c15bde0 0x7f7b4c15c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:34.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.910+0000 7f7b50870700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f7b4c15bde0 0x7f7b4c15c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58844/0 (socket says 172.21.15.91:58844) 2023-11-24T06:08:34.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.910+0000 7f7b50870700 1 -- 172.21.15.91:0/292431658 learned_addr learned my addr 172.21.15.91:0/292431658 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:34.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.910+0000 7f7b51071700 1 --2- 172.21.15.91:0/292431658 >> v2:172.21.15.94:3300/0 conn(0x7f7b4c0ce5c0 0x7f7b4c0cea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:34.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.910+0000 7f7b50870700 1 -- 172.21.15.91:0/292431658 >> v2:172.21.15.94:3300/0 conn(0x7f7b4c0ce5c0 msgr2=0x7f7b4c0cea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:34.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.910+0000 7f7b50870700 1 --2- 172.21.15.91:0/292431658 >> v2:172.21.15.94:3300/0 conn(0x7f7b4c0ce5c0 0x7f7b4c0cea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.910+0000 7f7b50870700 1 -- 172.21.15.91:0/292431658 >> v1:172.21.15.94:6789/0 conn(0x7f7b4c0cf2b0 legacy=0x7f7b4c0d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:34.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.910+0000 7f7b50870700 1 -- 172.21.15.91:0/292431658 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4c06f0f0 con 0x7f7b4c15bde0 2023-11-24T06:08:34.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.911+0000 7f7b50870700 1 --2- 172.21.15.91:0/292431658 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7b4c15bde0 0x7f7b4c15c1c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c068360 tx=0x7f7b40009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2dd02a84c42c976 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:34.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.911+0000 7f7b4b7fe700 1 -- 172.21.15.91:0/292431658 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b40011830 con 0x7f7b4c15bde0 2023-11-24T06:08:34.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.911+0000 7f7b4b7fe700 1 -- 172.21.15.91:0/292431658 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b40011990 con 0x7f7b4c15bde0 2023-11-24T06:08:34.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.911+0000 7f7b4b7fe700 1 -- 172.21.15.91:0/292431658 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b40011d20 con 0x7f7b4c15bde0 2023-11-24T06:08:34.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.911+0000 7f7b52ad4700 1 -- 172.21.15.91:0/292431658 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7b4c15bde0 msgr2=0x7f7b4c15c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:34.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.911+0000 7f7b52ad4700 1 --2- 172.21.15.91:0/292431658 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7b4c15bde0 0x7f7b4c15c1c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c068360 tx=0x7f7b40009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.911+0000 7f7b52ad4700 1 -- 172.21.15.91:0/292431658 shutdown_connections 2023-11-24T06:08:34.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.911+0000 7f7b52ad4700 1 --2- 172.21.15.91:0/292431658 >> v2:172.21.15.94:3300/0 conn(0x7f7b4c0ce5c0 0x7f7b4c0cea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.911+0000 7f7b52ad4700 1 --2- 172.21.15.91:0/292431658 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7b4c15bde0 0x7f7b4c15c1c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.911+0000 7f7b52ad4700 1 -- 172.21.15.91:0/292431658 >> 172.21.15.91:0/292431658 conn(0x7f7b4c0b3720 msgr2=0x7f7b4c0b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:34.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.911+0000 7f7b52ad4700 1 -- 172.21.15.91:0/292431658 shutdown_connections 2023-11-24T06:08:34.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.911+0000 7f7b52ad4700 1 -- 172.21.15.91:0/292431658 wait complete. 2023-11-24T06:08:34.919 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.912+0000 7f7b52ad4700 1 Processor -- start 2023-11-24T06:08:34.919 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.912+0000 7f7b52ad4700 1 -- start start 2023-11-24T06:08:34.919 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.913+0000 7f7b52ad4700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b4c0ce5c0 0x7f7b4c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:34.919 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.913+0000 7f7b52ad4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b4c0cf2b0 0x7f7b4c15d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:34.919 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.913+0000 7f7b52ad4700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7b4c15bde0 0x7f7b4c15dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:34.919 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.913+0000 7f7b52ad4700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c15d4d0 con 0x7f7b4c15bde0 2023-11-24T06:08:34.919 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.913+0000 7f7b52ad4700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c15e3d0 con 0x7f7b4c0ce5c0 2023-11-24T06:08:34.919 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.913+0000 7f7b52ad4700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c15e6b0 con 0x7f7b4c0cf2b0 2023-11-24T06:08:34.919 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.913+0000 7f7b51071700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7b4c15bde0 0x7f7b4c15dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:34.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.913+0000 7f7b51071700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7b4c15bde0 0x7f7b4c15dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58850/0 (socket says 172.21.15.91:58850) 2023-11-24T06:08:34.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.913+0000 7f7b51071700 1 -- 172.21.15.91:0/778827621 learned_addr learned my addr 172.21.15.91:0/778827621 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:34.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.914+0000 7f7b50870700 1 --2- 172.21.15.91:0/778827621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b4c0ce5c0 0x7f7b4c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:34.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.914+0000 7f7b4bfff700 1 --2- 172.21.15.91:0/778827621 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b4c0cf2b0 0x7f7b4c15d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:34.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.914+0000 7f7b51071700 1 -- 172.21.15.91:0/778827621 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b4c0cf2b0 msgr2=0x7f7b4c15d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:34.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.914+0000 7f7b51071700 1 --2- 172.21.15.91:0/778827621 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b4c0cf2b0 0x7f7b4c15d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.914+0000 7f7b51071700 1 -- 172.21.15.91:0/778827621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b4c0ce5c0 msgr2=0x7f7b4c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:34.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.914+0000 7f7b51071700 1 --2- 172.21.15.91:0/778827621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b4c0ce5c0 0x7f7b4c15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:34.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.914+0000 7f7b51071700 1 -- 172.21.15.91:0/778827621 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b40009550 con 0x7f7b4c15bde0 2023-11-24T06:08:34.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.914+0000 7f7b51071700 1 --2- 172.21.15.91:0/778827621 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7b4c15bde0 0x7f7b4c15dda0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7b4400efc0 tx=0x7f7b4400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:34.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.914+0000 7f7b497fa700 1 -- 172.21.15.91:0/778827621 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b44009ea0 con 0x7f7b4c15bde0 2023-11-24T06:08:34.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.914+0000 7f7b497fa700 1 -- 172.21.15.91:0/778827621 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4400f040 con 0x7f7b4c15bde0 2023-11-24T06:08:34.920 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.914+0000 7f7b52ad4700 1 -- 172.21.15.91:0/778827621 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4c15e990 con 0x7f7b4c15bde0 2023-11-24T06:08:34.921 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.914+0000 7f7b497fa700 1 -- 172.21.15.91:0/778827621 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4400ced0 con 0x7f7b4c15bde0 2023-11-24T06:08:34.921 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.914+0000 7f7b52ad4700 1 -- 172.21.15.91:0/778827621 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4c15ef50 con 0x7f7b4c15bde0 2023-11-24T06:08:34.921 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.915+0000 7f7b52ad4700 1 -- 172.21.15.91:0/778827621 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b4c075480 con 0x7f7b4c15bde0 2023-11-24T06:08:34.921 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.915+0000 7f7b497fa700 1 -- 172.21.15.91:0/778827621 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7b44014480 con 0x7f7b4c15bde0 2023-11-24T06:08:34.921 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.916+0000 7f7b497fa700 1 --2- 172.21.15.91:0/778827621 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7b3006aed0 0x7f7b3006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:34.921 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.916+0000 7f7b497fa700 1 -- 172.21.15.91:0/778827621 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b44089d90 con 0x7f7b4c15bde0 2023-11-24T06:08:34.921 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.916+0000 7f7b50870700 1 --2- 172.21.15.91:0/778827621 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7b3006aed0 0x7f7b3006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:34.921 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.916+0000 7f7b50870700 1 --2- 172.21.15.91:0/778827621 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7b3006aed0 0x7f7b3006d390 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7b40014040 tx=0x7f7b40008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:34.923 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:34.920+0000 7f7b497fa700 1 -- 172.21.15.91:0/778827621 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b44054e50 con 0x7f7b4c15bde0 2023-11-24T06:08:35.116 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.113+0000 7f7b52ad4700 1 -- 172.21.15.91:0/778827621 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f7b4c0cde90 con 0x7f7b4c15bde0 2023-11-24T06:08:35.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.116+0000 7f7b497fa700 1 -- 172.21.15.91:0/778827621 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7b44058470 con 0x7f7b4c15bde0 2023-11-24T06:08:35.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.118+0000 7f7b52ad4700 1 -- 172.21.15.91:0/778827621 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7b3006aed0 msgr2=0x7f7b3006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:35.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.119+0000 7f7b52ad4700 1 --2- 172.21.15.91:0/778827621 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7b3006aed0 0x7f7b3006d390 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7b40014040 tx=0x7f7b40008030 comp rx=0 tx=0).stop 2023-11-24T06:08:35.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.119+0000 7f7b52ad4700 1 -- 172.21.15.91:0/778827621 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7b4c15bde0 msgr2=0x7f7b4c15dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:35.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.119+0000 7f7b52ad4700 1 --2- 172.21.15.91:0/778827621 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7b4c15bde0 0x7f7b4c15dda0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7b4400efc0 tx=0x7f7b4400c3a0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.119+0000 7f7b52ad4700 1 -- 172.21.15.91:0/778827621 shutdown_connections 2023-11-24T06:08:35.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.119+0000 7f7b52ad4700 1 --2- 172.21.15.91:0/778827621 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7b3006aed0 0x7f7b3006d390 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.119+0000 7f7b52ad4700 1 --2- 172.21.15.91:0/778827621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7b4c0ce5c0 0x7f7b4c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.119+0000 7f7b52ad4700 1 --2- 172.21.15.91:0/778827621 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7b4c15bde0 0x7f7b4c15dda0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.119+0000 7f7b52ad4700 1 --2- 172.21.15.91:0/778827621 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7b4c0cf2b0 0x7f7b4c15d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.119+0000 7f7b52ad4700 1 -- 172.21.15.91:0/778827621 >> 172.21.15.91:0/778827621 conn(0x7f7b4c0b3720 msgr2=0x7f7b4c0b06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:35.123 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.119+0000 7f7b52ad4700 1 -- 172.21.15.91:0/778827621 shutdown_connections 2023-11-24T06:08:35.123 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.119+0000 7f7b52ad4700 1 -- 172.21.15.91:0/778827621 wait complete. 2023-11-24T06:08:35.134 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T06:08:35.134 DEBUG:teuthology.orchestra.run.smithi091:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T06:08:35.182 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T06:08:35.233 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T06:08:35.233 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:08:35.233 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:08:35.250 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:08:35.250 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:08:35.309 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:08:35.309 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:08:35.366 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:35.526 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.523+0000 7fb7c225e700 1 Processor -- start 2023-11-24T06:08:35.526 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.523+0000 7fb7c225e700 1 -- start start 2023-11-24T06:08:35.529 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.524+0000 7fb7c225e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7bc15dfc0 0x7fb7bc0d4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:35.529 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.524+0000 7fb7c225e700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb7bc0d1440 0x7fb7bc0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:35.529 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.524+0000 7fb7c225e700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fb7bc15e3a0 0x7fb7bc0d0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:35.529 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.524+0000 7fb7c225e700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7bc0d2790 con 0x7fb7bc15e3a0 2023-11-24T06:08:35.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.524+0000 7fb7c225e700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7bc0d2a70 con 0x7fb7bc0d1440 2023-11-24T06:08:35.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.524+0000 7fb7c225e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc0c7a30 con 0x7fb7bc15dfc0 2023-11-24T06:08:35.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.524+0000 7fb7bbfff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fb7bc15e3a0 0x7fb7bc0d0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:35.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.524+0000 7fb7bbfff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fb7bc15e3a0 0x7fb7bc0d0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58858/0 (socket says 172.21.15.91:58858) 2023-11-24T06:08:35.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.524+0000 7fb7bbfff700 1 -- 172.21.15.91:0/1989703860 learned_addr learned my addr 172.21.15.91:0/1989703860 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:35.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.525+0000 7fb7baffd700 1 --2- 172.21.15.91:0/1989703860 >> v2:172.21.15.94:3300/0 conn(0x7fb7bc0d1440 0x7fb7bc0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:35.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.525+0000 7fb7bbfff700 1 -- 172.21.15.91:0/1989703860 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7bc15dfc0 msgr2=0x7fb7bc0d4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:35.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.525+0000 7fb7bb7fe700 1 --2- 172.21.15.91:0/1989703860 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7bc15dfc0 0x7fb7bc0d4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:35.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.525+0000 7fb7bbfff700 1 --2- 172.21.15.91:0/1989703860 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7bc15dfc0 0x7fb7bc0d4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.525+0000 7fb7bbfff700 1 -- 172.21.15.91:0/1989703860 >> v2:172.21.15.94:3300/0 conn(0x7fb7bc0d1440 msgr2=0x7fb7bc0d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:35.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.525+0000 7fb7bbfff700 1 --2- 172.21.15.91:0/1989703860 >> v2:172.21.15.94:3300/0 conn(0x7fb7bc0d1440 0x7fb7bc0d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.525+0000 7fb7bbfff700 1 -- 172.21.15.91:0/1989703860 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7bc0c7d10 con 0x7fb7bc15e3a0 2023-11-24T06:08:35.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.525+0000 7fb7baffd700 1 --2- 172.21.15.91:0/1989703860 >> v2:172.21.15.94:3300/0 conn(0x7fb7bc0d1440 0x7fb7bc0d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:35.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.525+0000 7fb7bb7fe700 1 --2- 172.21.15.91:0/1989703860 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7bc15dfc0 0x7fb7bc0d4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:35.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.525+0000 7fb7bbfff700 1 --2- 172.21.15.91:0/1989703860 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7bc15e3a0 0x7fb7bc0d0d70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc08e630 tx=0x7fb7b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=286e3c0498945392 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:35.532 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.525+0000 7fb7ba7fc700 1 -- 172.21.15.91:0/1989703860 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b400e070 con 0x7fb7bc15e3a0 2023-11-24T06:08:35.532 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.525+0000 7fb7ba7fc700 1 -- 172.21.15.91:0/1989703860 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b400b9c0 con 0x7fb7bc15e3a0 2023-11-24T06:08:35.532 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.525+0000 7fb7ba7fc700 1 -- 172.21.15.91:0/1989703860 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b400bd70 con 0x7fb7bc15e3a0 2023-11-24T06:08:35.532 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.526+0000 7fb7c225e700 1 -- 172.21.15.91:0/1989703860 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7bc15e3a0 msgr2=0x7fb7bc0d0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:35.532 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.526+0000 7fb7c225e700 1 --2- 172.21.15.91:0/1989703860 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7bc15e3a0 0x7fb7bc0d0d70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc08e630 tx=0x7fb7b400b330 comp rx=0 tx=0).stop 2023-11-24T06:08:35.532 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.526+0000 7fb7c225e700 1 -- 172.21.15.91:0/1989703860 shutdown_connections 2023-11-24T06:08:35.532 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.526+0000 7fb7c225e700 1 --2- 172.21.15.91:0/1989703860 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7bc15e3a0 0x7fb7bc0d0d70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.532 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.526+0000 7fb7c225e700 1 --2- 172.21.15.91:0/1989703860 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7bc15dfc0 0x7fb7bc0d4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.533 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.526+0000 7fb7c225e700 1 --2- 172.21.15.91:0/1989703860 >> v2:172.21.15.94:3300/0 conn(0x7fb7bc0d1440 0x7fb7bc0d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.533 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.526+0000 7fb7c225e700 1 -- 172.21.15.91:0/1989703860 >> 172.21.15.91:0/1989703860 conn(0x7fb7bc0d3e40 msgr2=0x7fb7bc0d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:35.533 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.526+0000 7fb7c225e700 1 -- 172.21.15.91:0/1989703860 shutdown_connections 2023-11-24T06:08:35.533 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.526+0000 7fb7c225e700 1 -- 172.21.15.91:0/1989703860 wait complete. 2023-11-24T06:08:35.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.527+0000 7fb7c225e700 1 Processor -- start 2023-11-24T06:08:35.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.527+0000 7fb7c225e700 1 -- start start 2023-11-24T06:08:35.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7c225e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7bc0d1440 0x7fb7bc152250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:35.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7c225e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7bc15dfc0 0x7fb7bc0cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:35.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7c225e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7bc15e3a0 0x7fb7bc0b87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:35.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7c225e700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc15b7d0 con 0x7fb7bc15dfc0 2023-11-24T06:08:35.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7c225e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc152790 con 0x7fb7bc15e3a0 2023-11-24T06:08:35.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7c225e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc0cb2d0 con 0x7fb7bc0d1440 2023-11-24T06:08:35.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7baffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7bc15dfc0 0x7fb7bc0cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:35.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7baffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7bc15dfc0 0x7fb7bc0cc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58862/0 (socket says 172.21.15.91:58862) 2023-11-24T06:08:35.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7baffd700 1 -- 172.21.15.91:0/1448197164 learned_addr learned my addr 172.21.15.91:0/1448197164 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:35.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7bbfff700 1 --2- 172.21.15.91:0/1448197164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7bc15e3a0 0x7fb7bc0b87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:35.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7bb7fe700 1 --2- 172.21.15.91:0/1448197164 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7bc0d1440 0x7fb7bc152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:35.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7baffd700 1 -- 172.21.15.91:0/1448197164 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7bc0d1440 msgr2=0x7fb7bc152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:35.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7baffd700 1 --2- 172.21.15.91:0/1448197164 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7bc0d1440 0x7fb7bc152250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7baffd700 1 -- 172.21.15.91:0/1448197164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7bc15e3a0 msgr2=0x7fb7bc0b87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:35.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7baffd700 1 --2- 172.21.15.91:0/1448197164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7bc15e3a0 0x7fb7bc0b87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7baffd700 1 -- 172.21.15.91:0/1448197164 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b4009b30 con 0x7fb7bc15dfc0 2023-11-24T06:08:35.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7baffd700 1 --2- 172.21.15.91:0/1448197164 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7bc15dfc0 0x7fb7bc0cc370 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac00c750 tx=0x7fb7ac00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:35.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.528+0000 7fb7b3fff700 1 -- 172.21.15.91:0/1448197164 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac011800 con 0x7fb7bc15dfc0 2023-11-24T06:08:35.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.529+0000 7fb7b3fff700 1 -- 172.21.15.91:0/1448197164 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7ac011960 con 0x7fb7bc15dfc0 2023-11-24T06:08:35.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.529+0000 7fb7c225e700 1 -- 172.21.15.91:0/1448197164 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7bc0cb5b0 con 0x7fb7bc15dfc0 2023-11-24T06:08:35.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.529+0000 7fb7b3fff700 1 -- 172.21.15.91:0/1448197164 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7ac011cd0 con 0x7fb7bc15dfc0 2023-11-24T06:08:35.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.529+0000 7fb7c225e700 1 -- 172.21.15.91:0/1448197164 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7bc0cbb70 con 0x7fb7bc15dfc0 2023-11-24T06:08:35.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.529+0000 7fb7c225e700 1 -- 172.21.15.91:0/1448197164 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7bc068670 con 0x7fb7bc15dfc0 2023-11-24T06:08:35.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.530+0000 7fb7b3fff700 1 -- 172.21.15.91:0/1448197164 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb7ac021350 con 0x7fb7bc15dfc0 2023-11-24T06:08:35.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.530+0000 7fb7b3fff700 1 --2- 172.21.15.91:0/1448197164 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb79c06aed0 0x7fb79c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:35.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.530+0000 7fb7b3fff700 1 -- 172.21.15.91:0/1448197164 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7ac089600 con 0x7fb7bc15dfc0 2023-11-24T06:08:35.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.530+0000 7fb7bb7fe700 1 --2- 172.21.15.91:0/1448197164 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb79c06aed0 0x7fb79c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:35.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.531+0000 7fb7bb7fe700 1 --2- 172.21.15.91:0/1448197164 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb79c06aed0 0x7fb79c06d390 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4006fd0 tx=0x7fb7a4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:35.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.535+0000 7fb7b3fff700 1 -- 172.21.15.91:0/1448197164 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb7ac054630 con 0x7fb7bc15dfc0 2023-11-24T06:08:35.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.724+0000 7fb7c225e700 1 -- 172.21.15.91:0/1448197164 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7bc08d550 con 0x7fb7bc15dfc0 2023-11-24T06:08:35.728 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.725+0000 7fb7b3fff700 1 -- 172.21.15.91:0/1448197164 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7fb7ac057c50 con 0x7fb7bc15dfc0 2023-11-24T06:08:35.728 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:35.729 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":20,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:02.415612+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:35.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.728+0000 7fb7c225e700 1 -- 172.21.15.91:0/1448197164 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb79c06aed0 msgr2=0x7fb79c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:35.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.728+0000 7fb7c225e700 1 --2- 172.21.15.91:0/1448197164 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb79c06aed0 0x7fb79c06d390 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4006fd0 tx=0x7fb7a4008040 comp rx=0 tx=0).stop 2023-11-24T06:08:35.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.728+0000 7fb7c225e700 1 -- 172.21.15.91:0/1448197164 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7bc15dfc0 msgr2=0x7fb7bc0cc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:35.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.728+0000 7fb7c225e700 1 --2- 172.21.15.91:0/1448197164 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7bc15dfc0 0x7fb7bc0cc370 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac00c750 tx=0x7fb7ac00cb10 comp rx=0 tx=0).stop 2023-11-24T06:08:35.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.728+0000 7fb7c225e700 1 -- 172.21.15.91:0/1448197164 shutdown_connections 2023-11-24T06:08:35.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.728+0000 7fb7c225e700 1 --2- 172.21.15.91:0/1448197164 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb79c06aed0 0x7fb79c06d390 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.728+0000 7fb7c225e700 1 --2- 172.21.15.91:0/1448197164 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb7bc15e3a0 0x7fb7bc0b87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.728+0000 7fb7c225e700 1 --2- 172.21.15.91:0/1448197164 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb7bc0d1440 0x7fb7bc152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.728+0000 7fb7c225e700 1 --2- 172.21.15.91:0/1448197164 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb7bc15dfc0 0x7fb7bc0cc370 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.732 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.728+0000 7fb7c225e700 1 -- 172.21.15.91:0/1448197164 >> 172.21.15.91:0/1448197164 conn(0x7fb7bc0d3e40 msgr2=0x7fb7bc0c8340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:35.732 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.728+0000 7fb7c225e700 1 -- 172.21.15.91:0/1448197164 shutdown_connections 2023-11-24T06:08:35.732 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.728+0000 7fb7c225e700 1 -- 172.21.15.91:0/1448197164 wait complete. 2023-11-24T06:08:35.743 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:07:58.710106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:07:59.774819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:08:35.744 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:08:35.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.865+0000 7f6d6e0df700 1 Processor -- start 2023-11-24T06:08:35.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.865+0000 7f6d6e0df700 1 -- start start 2023-11-24T06:08:35.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d6e0df700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6d680b4150 0x7f6d6806de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:35.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d6e0df700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d68071f50 con 0x7f6d6806b620 2023-11-24T06:08:35.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d6e0df700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d68072230 con 0x7f6d680b5ab0 2023-11-24T06:08:35.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d6e0df700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d68072510 con 0x7f6d680b4150 2023-11-24T06:08:35.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d677fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f6d6806b620 0x7f6d680b3a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53802/0 (socket says 172.21.15.91:53802) 2023-11-24T06:08:35.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d677fe700 1 -- 172.21.15.91:0/1580966093 learned_addr learned my addr 172.21.15.91:0/1580966093 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:35.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d66ffd700 1 --2- 172.21.15.91:0/1580966093 >> v2:172.21.15.94:3300/0 conn(0x7f6d680b4150 0x7f6d6806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:35.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d667fc700 1 -- 172.21.15.91:0/1580966093 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075837964 0 0) 0x7f6d68071f50 con 0x7f6d6806b620 2023-11-24T06:08:35.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d667fc700 1 -- 172.21.15.91:0/1580966093 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d540035b0 con 0x7f6d6806b620 2023-11-24T06:08:35.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d667fc700 1 -- 172.21.15.91:0/1580966093 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 343987255 0 0) 0x7f6d540035b0 con 0x7f6d6806b620 2023-11-24T06:08:35.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d667fc700 1 -- 172.21.15.91:0/1580966093 >> v2:172.21.15.94:3300/0 conn(0x7f6d680b4150 msgr2=0x7f6d6806de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:35.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d667fc700 1 --2- 172.21.15.91:0/1580966093 >> v2:172.21.15.94:3300/0 conn(0x7f6d680b4150 0x7f6d6806de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d667fc700 1 -- 172.21.15.91:0/1580966093 >> v1:172.21.15.94:6789/0 conn(0x7f6d680b5ab0 legacy=0x7f6d6806e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:35.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d667fc700 1 -- 172.21.15.91:0/1580966093 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d680727f0 con 0x7f6d6806b620 2023-11-24T06:08:35.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d667fc700 1 -- 172.21.15.91:0/1580966093 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f6d50001670 con 0x7f6d6806b620 2023-11-24T06:08:35.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d667fc700 1 -- 172.21.15.91:0/1580966093 >> v1:172.21.15.91:6789/0 conn(0x7f6d6806b620 legacy=0x7f6d680b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:35.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d667fc700 1 --2- 172.21.15.91:0/1580966093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d54003a00 0x7f6d54003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:35.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d667fc700 1 --2- 172.21.15.91:0/1580966093 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6d54004850 0x7f6d54004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:35.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d667fc700 1 --2- 172.21.15.91:0/1580966093 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6d54005210 0x7f6d5400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:35.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.866+0000 7f6d667fc700 1 -- 172.21.15.91:0/1580966093 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d68072510 con 0x7f6d54004850 2023-11-24T06:08:35.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.867+0000 7f6d667fc700 1 -- 172.21.15.91:0/1580966093 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5400c0b0 con 0x7f6d54003a00 2023-11-24T06:08:35.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.867+0000 7f6d667fc700 1 -- 172.21.15.91:0/1580966093 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5400c270 con 0x7f6d54005210 2023-11-24T06:08:35.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.867+0000 7f6d66ffd700 1 --2- 172.21.15.91:0/1580966093 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6d54004850 0x7f6d54004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:35.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.867+0000 7f6d677fe700 1 --2- 172.21.15.91:0/1580966093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d54003a00 0x7f6d54003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:35.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.867+0000 7f6d67fff700 1 --2- 172.21.15.91:0/1580966093 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6d54005210 0x7f6d5400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:35.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.867+0000 7f6d66ffd700 1 -- 172.21.15.91:0/1580966093 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6d54005210 msgr2=0x7f6d5400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:35.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.867+0000 7f6d66ffd700 1 --2- 172.21.15.91:0/1580966093 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6d54005210 0x7f6d5400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.867+0000 7f6d66ffd700 1 -- 172.21.15.91:0/1580966093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d54003a00 msgr2=0x7f6d54003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:35.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.867+0000 7f6d66ffd700 1 --2- 172.21.15.91:0/1580966093 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d54003a00 0x7f6d54003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.867+0000 7f6d66ffd700 1 -- 172.21.15.91:0/1580966093 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d540035b0 con 0x7f6d54004850 2023-11-24T06:08:35.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.867+0000 7f6d67fff700 1 --2- 172.21.15.91:0/1580966093 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6d54005210 0x7f6d5400ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:35.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.867+0000 7f6d67fff700 1 -- 172.21.15.91:0/1580966093 reap_dead start 2023-11-24T06:08:35.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.867+0000 7f6d66ffd700 1 --2- 172.21.15.91:0/1580966093 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6d54004850 0x7f6d54004cd0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009b60 tx=0x7f6d54005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f671c9f57744678 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:35.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.867+0000 7f6d667fc700 1 -- 172.21.15.91:0/1580966093 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5800e6c0 con 0x7f6d54004850 2023-11-24T06:08:35.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.867+0000 7f6d667fc700 1 -- 172.21.15.91:0/1580966093 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5800e820 con 0x7f6d54004850 2023-11-24T06:08:35.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.867+0000 7f6d667fc700 1 -- 172.21.15.91:0/1580966093 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5800eb90 con 0x7f6d54004850 2023-11-24T06:08:35.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.868+0000 7f6d6e0df700 1 -- 172.21.15.91:0/1580966093 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6d54004850 msgr2=0x7f6d54004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:35.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.868+0000 7f6d6e0df700 1 --2- 172.21.15.91:0/1580966093 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6d54004850 0x7f6d54004cd0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009b60 tx=0x7f6d54005880 comp rx=0 tx=0).stop 2023-11-24T06:08:35.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.868+0000 7f6d6e0df700 1 -- 172.21.15.91:0/1580966093 shutdown_connections 2023-11-24T06:08:35.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.868+0000 7f6d6e0df700 1 --2- 172.21.15.91:0/1580966093 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6d54004850 0x7f6d54004cd0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.868+0000 7f6d6e0df700 1 -- 172.21.15.91:0/1580966093 >> 172.21.15.91:0/1580966093 conn(0x7f6d680b2740 msgr2=0x7f6d680b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:35.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.868+0000 7f6d6e0df700 1 -- 172.21.15.91:0/1580966093 shutdown_connections 2023-11-24T06:08:35.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.868+0000 7f6d6e0df700 1 -- 172.21.15.91:0/1580966093 wait complete. 2023-11-24T06:08:35.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.869+0000 7f6d6e0df700 1 Processor -- start 2023-11-24T06:08:35.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.870+0000 7f6d6e0df700 1 -- start start 2023-11-24T06:08:35.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.870+0000 7f6d6e0df700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6d54004850 0x7f6d681f6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:35.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.870+0000 7f6d6e0df700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d6806eea0 0x7f6d6806f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:35.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.870+0000 7f6d6e0df700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6d680c76c0 0x7f6d680c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:35.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.870+0000 7f6d6e0df700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6806f860 con 0x7f6d680c76c0 2023-11-24T06:08:35.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.870+0000 7f6d6e0df700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d680c83b0 con 0x7f6d6806eea0 2023-11-24T06:08:35.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.870+0000 7f6d6e0df700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d680c8690 con 0x7f6d54004850 2023-11-24T06:08:35.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d67fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6d680c76c0 0x7f6d680c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:35.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d67fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6d680c76c0 0x7f6d680c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58884/0 (socket says 172.21.15.91:58884) 2023-11-24T06:08:35.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d67fff700 1 -- 172.21.15.91:0/3895260244 learned_addr learned my addr 172.21.15.91:0/3895260244 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:35.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d677fe700 1 --2- 172.21.15.91:0/3895260244 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6d54004850 0x7f6d681f6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:35.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d66ffd700 1 --2- 172.21.15.91:0/3895260244 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d6806eea0 0x7f6d6806f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:35.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d67fff700 1 -- 172.21.15.91:0/3895260244 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6d54004850 msgr2=0x7f6d681f6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:35.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d67fff700 1 --2- 172.21.15.91:0/3895260244 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6d54004850 0x7f6d681f6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d67fff700 1 -- 172.21.15.91:0/3895260244 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d6806eea0 msgr2=0x7f6d6806f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:35.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d67fff700 1 --2- 172.21.15.91:0/3895260244 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d6806eea0 0x7f6d6806f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:35.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d67fff700 1 -- 172.21.15.91:0/3895260244 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d680c8970 con 0x7f6d680c76c0 2023-11-24T06:08:35.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d66ffd700 1 --2- 172.21.15.91:0/3895260244 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d6806eea0 0x7f6d6806f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:35.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d67fff700 1 --2- 172.21.15.91:0/3895260244 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6d680c76c0 0x7f6d680c7b40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6d68073050 tx=0x7f6d6000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:35.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d5ffff700 1 -- 172.21.15.91:0/3895260244 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60003c10 con 0x7f6d680c76c0 2023-11-24T06:08:35.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d5ffff700 1 -- 172.21.15.91:0/3895260244 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d60003d90 con 0x7f6d680c76c0 2023-11-24T06:08:35.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d6e0df700 1 -- 172.21.15.91:0/3895260244 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d680c8c50 con 0x7f6d680c76c0 2023-11-24T06:08:35.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d5ffff700 1 -- 172.21.15.91:0/3895260244 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60005310 con 0x7f6d680c76c0 2023-11-24T06:08:35.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.871+0000 7f6d6e0df700 1 -- 172.21.15.91:0/3895260244 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d680c8080 con 0x7f6d680c76c0 2023-11-24T06:08:35.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.872+0000 7f6d6e0df700 1 -- 172.21.15.91:0/3895260244 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d68075480 con 0x7f6d680c76c0 2023-11-24T06:08:35.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.873+0000 7f6d5ffff700 1 -- 172.21.15.91:0/3895260244 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6d60005470 con 0x7f6d680c76c0 2023-11-24T06:08:35.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.873+0000 7f6d5ffff700 1 --2- 172.21.15.91:0/3895260244 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6d4806aed0 0x7f6d4806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:35.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.873+0000 7f6d5ffff700 1 -- 172.21.15.91:0/3895260244 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d600894b0 con 0x7f6d680c76c0 2023-11-24T06:08:35.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.873+0000 7f6d677fe700 1 --2- 172.21.15.91:0/3895260244 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6d4806aed0 0x7f6d4806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:35.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.874+0000 7f6d677fe700 1 --2- 172.21.15.91:0/3895260244 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6d4806aed0 0x7f6d4806d390 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6d50007db0 tx=0x7f6d50009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:35.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:35.878+0000 7f6d5ffff700 1 -- 172.21.15.91:0/3895260244 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d600545f0 con 0x7f6d680c76c0 2023-11-24T06:08:36.069 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.066+0000 7f6d6e0df700 1 -- 172.21.15.91:0/3895260244 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6d680b0a10 con 0x7f6d680c76c0 2023-11-24T06:08:36.069 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.066+0000 7f6d5ffff700 1 -- 172.21.15.91:0/3895260244 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6d60057c10 con 0x7f6d680c76c0 2023-11-24T06:08:36.069 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:08:36.071 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.068+0000 7f6d6e0df700 1 -- 172.21.15.91:0/3895260244 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6d4806aed0 msgr2=0x7f6d4806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.071 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.068+0000 7f6d6e0df700 1 --2- 172.21.15.91:0/3895260244 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6d4806aed0 0x7f6d4806d390 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6d50007db0 tx=0x7f6d50009040 comp rx=0 tx=0).stop 2023-11-24T06:08:36.071 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.068+0000 7f6d6e0df700 1 -- 172.21.15.91:0/3895260244 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6d680c76c0 msgr2=0x7f6d680c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.071 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.068+0000 7f6d6e0df700 1 --2- 172.21.15.91:0/3895260244 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6d680c76c0 0x7f6d680c7b40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6d68073050 tx=0x7f6d6000ee00 comp rx=0 tx=0).stop 2023-11-24T06:08:36.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.068+0000 7f6d6e0df700 1 -- 172.21.15.91:0/3895260244 shutdown_connections 2023-11-24T06:08:36.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.068+0000 7f6d6e0df700 1 --2- 172.21.15.91:0/3895260244 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6d4806aed0 0x7f6d4806d390 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.068+0000 7f6d6e0df700 1 --2- 172.21.15.91:0/3895260244 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6d680c76c0 0x7f6d680c7b40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.068+0000 7f6d6e0df700 1 --2- 172.21.15.91:0/3895260244 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6d54004850 0x7f6d681f6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.068+0000 7f6d6e0df700 1 --2- 172.21.15.91:0/3895260244 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6d6806eea0 0x7f6d6806f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.068+0000 7f6d6e0df700 1 -- 172.21.15.91:0/3895260244 >> 172.21.15.91:0/3895260244 conn(0x7f6d680b2740 msgr2=0x7f6d68146220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:36.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.068+0000 7f6d6e0df700 1 -- 172.21.15.91:0/3895260244 shutdown_connections 2023-11-24T06:08:36.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.068+0000 7f6d6e0df700 1 -- 172.21.15.91:0/3895260244 wait complete. 2023-11-24T06:08:36.083 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:08:36.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.203+0000 7f94203b9700 1 Processor -- start 2023-11-24T06:08:36.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.203+0000 7f94203b9700 1 -- start start 2023-11-24T06:08:36.209 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f94203b9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f94180cf2b0 0x7f941815c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.209 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f94203b9700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f94180ce5c0 0x7f94180cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.209 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f94203b9700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94180d0a90 con 0x7f941815bde0 2023-11-24T06:08:36.209 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f94203b9700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94180cefb0 con 0x7f94180ce5c0 2023-11-24T06:08:36.209 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f94203b9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f941806ee10 con 0x7f94180cf2b0 2023-11-24T06:08:36.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f941e155700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f941815bde0 0x7f94180d01b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53822/0 (socket says 172.21.15.91:53822) 2023-11-24T06:08:36.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f941e155700 1 -- 172.21.15.91:0/2364269012 learned_addr learned my addr 172.21.15.91:0/2364269012 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:36.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f941d954700 1 --2- 172.21.15.91:0/2364269012 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f94180cf2b0 0x7f941815c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f941e956700 1 --2- 172.21.15.91:0/2364269012 >> v2:172.21.15.94:3300/0 conn(0x7f94180ce5c0 0x7f94180cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f941d153700 1 -- 172.21.15.91:0/2364269012 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296991266 0 0) 0x7f94180d0a90 con 0x7f941815bde0 2023-11-24T06:08:36.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f941d153700 1 -- 172.21.15.91:0/2364269012 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94040035b0 con 0x7f941815bde0 2023-11-24T06:08:36.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f941d153700 1 -- 172.21.15.91:0/2364269012 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1230744291 0 0) 0x7f94040035b0 con 0x7f941815bde0 2023-11-24T06:08:36.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f941d153700 1 -- 172.21.15.91:0/2364269012 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f94180cf2b0 msgr2=0x7f941815c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:36.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f941d153700 1 --2- 172.21.15.91:0/2364269012 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f94180cf2b0 0x7f941815c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f941d153700 1 -- 172.21.15.91:0/2364269012 >> v2:172.21.15.94:3300/0 conn(0x7f94180ce5c0 msgr2=0x7f94180cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:36.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f941d153700 1 --2- 172.21.15.91:0/2364269012 >> v2:172.21.15.94:3300/0 conn(0x7f94180ce5c0 0x7f94180cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f941d153700 1 -- 172.21.15.91:0/2364269012 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941806f0f0 con 0x7f941815bde0 2023-11-24T06:08:36.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f941d153700 1 -- 172.21.15.91:0/2364269012 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f9414001670 con 0x7f941815bde0 2023-11-24T06:08:36.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.204+0000 7f941d153700 1 -- 172.21.15.91:0/2364269012 >> v1:172.21.15.91:6789/0 conn(0x7f941815bde0 legacy=0x7f94180d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941d153700 1 --2- 172.21.15.91:0/2364269012 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f94180cf2b0 0x7f9404003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941d153700 1 --2- 172.21.15.91:0/2364269012 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9404003f80 0x7f9404004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941d153700 1 --2- 172.21.15.91:0/2364269012 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9404004d90 0x7f9404005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941d153700 1 -- 172.21.15.91:0/2364269012 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94180cefb0 con 0x7f9404004d90 2023-11-24T06:08:36.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941d153700 1 -- 172.21.15.91:0/2364269012 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941806ee10 con 0x7f9404003f80 2023-11-24T06:08:36.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941d153700 1 -- 172.21.15.91:0/2364269012 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940400ba60 con 0x7f94180cf2b0 2023-11-24T06:08:36.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941e956700 1 --2- 172.21.15.91:0/2364269012 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9404004d90 0x7f9404005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941e155700 1 --2- 172.21.15.91:0/2364269012 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f94180cf2b0 0x7f9404003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941d954700 1 --2- 172.21.15.91:0/2364269012 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9404003f80 0x7f9404004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941e956700 1 -- 172.21.15.91:0/2364269012 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f94180cf2b0 msgr2=0x7f9404003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:36.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941e956700 1 --2- 172.21.15.91:0/2364269012 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f94180cf2b0 0x7f9404003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941e956700 1 -- 172.21.15.91:0/2364269012 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9404003f80 msgr2=0x7f9404004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:36.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941e956700 1 --2- 172.21.15.91:0/2364269012 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f9404003f80 0x7f9404004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941e956700 1 -- 172.21.15.91:0/2364269012 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94040035b0 con 0x7f9404004d90 2023-11-24T06:08:36.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941e956700 1 --2- 172.21.15.91:0/2364269012 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9404004d90 0x7f9404005210 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f941000e170 tx=0x7f9404005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeb900c8ace49a76 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:36.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941d153700 1 -- 172.21.15.91:0/2364269012 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941000ec90 con 0x7f9404004d90 2023-11-24T06:08:36.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941d153700 1 -- 172.21.15.91:0/2364269012 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941000f040 con 0x7f9404004d90 2023-11-24T06:08:36.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.205+0000 7f941d153700 1 -- 172.21.15.91:0/2364269012 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9410014480 con 0x7f9404004d90 2023-11-24T06:08:36.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.206+0000 7f94203b9700 1 -- 172.21.15.91:0/2364269012 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9404004d90 msgr2=0x7f9404005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.206+0000 7f94203b9700 1 --2- 172.21.15.91:0/2364269012 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f9404004d90 0x7f9404005210 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f941000e170 tx=0x7f9404005880 comp rx=0 tx=0).stop 2023-11-24T06:08:36.212 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.206+0000 7f941e956700 1 -- 172.21.15.91:0/2364269012 reap_dead start 2023-11-24T06:08:36.213 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.206+0000 7f94203b9700 1 -- 172.21.15.91:0/2364269012 shutdown_connections 2023-11-24T06:08:36.213 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.206+0000 7f94203b9700 1 -- 172.21.15.91:0/2364269012 >> 172.21.15.91:0/2364269012 conn(0x7f94180b3720 msgr2=0x7f94180b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:36.213 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.206+0000 7f94203b9700 1 -- 172.21.15.91:0/2364269012 shutdown_connections 2023-11-24T06:08:36.213 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.206+0000 7f94203b9700 1 -- 172.21.15.91:0/2364269012 wait complete. 2023-11-24T06:08:36.215 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.207+0000 7f94203b9700 1 Processor -- start 2023-11-24T06:08:36.215 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.207+0000 7f94203b9700 1 -- start start 2023-11-24T06:08:36.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f94203b9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9404004d90 0x7f9418148dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f94203b9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f941814bfe0 0x7f94180cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f94203b9700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94180c8ca0 0x7f94180c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f94203b9700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94180c9660 con 0x7f94180c8ca0 2023-11-24T06:08:36.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f94203b9700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94180cbee0 con 0x7f941814bfe0 2023-11-24T06:08:36.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f94203b9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94180cb050 con 0x7f9404004d90 2023-11-24T06:08:36.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f941e956700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94180c8ca0 0x7f94180c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f941e956700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94180c8ca0 0x7f94180c9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58904/0 (socket says 172.21.15.91:58904) 2023-11-24T06:08:36.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f941e956700 1 -- 172.21.15.91:0/121938927 learned_addr learned my addr 172.21.15.91:0/121938927 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:36.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f941d954700 1 --2- 172.21.15.91:0/121938927 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f941814bfe0 0x7f94180cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.216 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f941e155700 1 --2- 172.21.15.91:0/121938927 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9404004d90 0x7f9418148dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f941e956700 1 -- 172.21.15.91:0/121938927 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9404004d90 msgr2=0x7f9418148dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f941e956700 1 --2- 172.21.15.91:0/121938927 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9404004d90 0x7f9418148dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f941e956700 1 -- 172.21.15.91:0/121938927 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f941814bfe0 msgr2=0x7f94180cab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f941e956700 1 --2- 172.21.15.91:0/121938927 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f941814bfe0 0x7f94180cab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f941e956700 1 -- 172.21.15.91:0/121938927 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94180cb2d0 con 0x7f94180c8ca0 2023-11-24T06:08:36.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f941e956700 1 --2- 172.21.15.91:0/121938927 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94180c8ca0 0x7f94180c9120 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9410016000 tx=0x7f9410004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:36.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f940effd700 1 -- 172.21.15.91:0/121938927 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9410007000 con 0x7f94180c8ca0 2023-11-24T06:08:36.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f940effd700 1 -- 172.21.15.91:0/121938927 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941000ee60 con 0x7f94180c8ca0 2023-11-24T06:08:36.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f940effd700 1 -- 172.21.15.91:0/121938927 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941001e670 con 0x7f94180c8ca0 2023-11-24T06:08:36.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.208+0000 7f94203b9700 1 -- 172.21.15.91:0/121938927 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94180cb5b0 con 0x7f94180c8ca0 2023-11-24T06:08:36.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.209+0000 7f94203b9700 1 -- 172.21.15.91:0/121938927 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94180cbb70 con 0x7f94180c8ca0 2023-11-24T06:08:36.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.209+0000 7f94203b9700 1 -- 172.21.15.91:0/121938927 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9418149500 con 0x7f94180c8ca0 2023-11-24T06:08:36.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.210+0000 7f940effd700 1 -- 172.21.15.91:0/121938927 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f9410024070 con 0x7f94180c8ca0 2023-11-24T06:08:36.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.210+0000 7f940effd700 1 --2- 172.21.15.91:0/121938927 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93fc06aed0 0x7f93fc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.210+0000 7f941e155700 1 --2- 172.21.15.91:0/121938927 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93fc06aed0 0x7f93fc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.210+0000 7f940effd700 1 -- 172.21.15.91:0/121938927 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f941008aaf0 con 0x7f94180c8ca0 2023-11-24T06:08:36.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.211+0000 7f941e155700 1 --2- 172.21.15.91:0/121938927 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93fc06aed0 0x7f93fc06d390 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f94140015d0 tx=0x7f9414007980 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:36.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.215+0000 7f940effd700 1 -- 172.21.15.91:0/121938927 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9418149500 con 0x7f94180c8ca0 2023-11-24T06:08:36.407 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.405+0000 7f94203b9700 1 -- 172.21.15.91:0/121938927 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9418149500 con 0x7f94180c8ca0 2023-11-24T06:08:36.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.405+0000 7f940effd700 1 -- 172.21.15.91:0/121938927 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9418149500 con 0x7f94180c8ca0 2023-11-24T06:08:36.408 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 32 2023-11-24T06:08:36.410 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.407+0000 7f94203b9700 1 -- 172.21.15.91:0/121938927 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93fc06aed0 msgr2=0x7f93fc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.410 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.407+0000 7f94203b9700 1 --2- 172.21.15.91:0/121938927 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93fc06aed0 0x7f93fc06d390 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f94140015d0 tx=0x7f9414007980 comp rx=0 tx=0).stop 2023-11-24T06:08:36.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.407+0000 7f94203b9700 1 -- 172.21.15.91:0/121938927 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94180c8ca0 msgr2=0x7f94180c9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.407+0000 7f94203b9700 1 --2- 172.21.15.91:0/121938927 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94180c8ca0 0x7f94180c9120 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9410016000 tx=0x7f9410004b40 comp rx=0 tx=0).stop 2023-11-24T06:08:36.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.407+0000 7f94203b9700 1 -- 172.21.15.91:0/121938927 shutdown_connections 2023-11-24T06:08:36.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.407+0000 7f94203b9700 1 --2- 172.21.15.91:0/121938927 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93fc06aed0 0x7f93fc06d390 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.407+0000 7f94203b9700 1 --2- 172.21.15.91:0/121938927 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94180c8ca0 0x7f94180c9120 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.407+0000 7f94203b9700 1 --2- 172.21.15.91:0/121938927 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f9404004d90 0x7f9418148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.407+0000 7f94203b9700 1 --2- 172.21.15.91:0/121938927 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f941814bfe0 0x7f94180cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.407+0000 7f94203b9700 1 -- 172.21.15.91:0/121938927 >> 172.21.15.91:0/121938927 conn(0x7f94180b3720 msgr2=0x7f94180cd990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:36.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.407+0000 7f94203b9700 1 -- 172.21.15.91:0/121938927 shutdown_connections 2023-11-24T06:08:36.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.407+0000 7f94203b9700 1 -- 172.21.15.91:0/121938927 wait complete. 2023-11-24T06:08:36.422 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:08:36.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.542+0000 7fc37749f700 1 Processor -- start 2023-11-24T06:08:36.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.542+0000 7fc37749f700 1 -- start start 2023-11-24T06:08:36.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.543+0000 7fc37749f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc370143ae0 0x7fc3700b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.543+0000 7fc37749f700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc3700722c0 0x7fc37006e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.543+0000 7fc37749f700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc370143ec0 con 0x7fc3701440f0 2023-11-24T06:08:36.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.543+0000 7fc37749f700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3700b40a0 con 0x7fc3700722c0 2023-11-24T06:08:36.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.543+0000 7fc37749f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3700b3ae0 con 0x7fc370143ae0 2023-11-24T06:08:36.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.543+0000 7fc37523b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc370143ae0 0x7fc3700b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.543+0000 7fc375a3c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc3700722c0 0x7fc37006e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.543+0000 7fc375a3c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc3700722c0 0x7fc37006e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:43304/0 (socket says 172.21.15.91:43304) 2023-11-24T06:08:36.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.543+0000 7fc37523b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc370143ae0 0x7fc3700b51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:42340/0 (socket says 172.21.15.91:42340) 2023-11-24T06:08:36.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.543+0000 7fc374a3a700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fc3701440f0 0x7fc370071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.91:55636/0 (socket says 172.21.15.91:55636) 2023-11-24T06:08:36.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.543+0000 7fc375a3c700 1 -- 172.21.15.91:0/1380064991 learned_addr learned my addr 172.21.15.91:0/1380064991 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:36.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.543+0000 7fc375a3c700 1 -- 172.21.15.91:0/1380064991 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc370143ae0 msgr2=0x7fc3700b51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:36.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.543+0000 7fc375a3c700 1 --2- 172.21.15.91:0/1380064991 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc370143ae0 0x7fc3700b51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.543+0000 7fc375a3c700 1 -- 172.21.15.91:0/1380064991 >> v1:172.21.15.94:6789/0 conn(0x7fc3701440f0 legacy=0x7fc370071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.543+0000 7fc375a3c700 1 -- 172.21.15.91:0/1380064991 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3700b3dc0 con 0x7fc3700722c0 2023-11-24T06:08:36.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.543+0000 7fc37523b700 1 --2- 172.21.15.91:0/1380064991 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc370143ae0 0x7fc3700b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:08:36.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.544+0000 7fc375a3c700 1 --2- 172.21.15.91:0/1380064991 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3700722c0 0x7fc37006e510 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc368009e50 tx=0x7fc36800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44766607e070a17d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:36.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.544+0000 7fc367fff700 1 -- 172.21.15.91:0/1380064991 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36800e070 con 0x7fc3700722c0 2023-11-24T06:08:36.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.544+0000 7fc367fff700 1 -- 172.21.15.91:0/1380064991 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36800b910 con 0x7fc3700722c0 2023-11-24T06:08:36.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.544+0000 7fc367fff700 1 -- 172.21.15.91:0/1380064991 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36800bcc0 con 0x7fc3700722c0 2023-11-24T06:08:36.550 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.544+0000 7fc37749f700 1 -- 172.21.15.91:0/1380064991 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3700722c0 msgr2=0x7fc37006e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.544+0000 7fc37749f700 1 --2- 172.21.15.91:0/1380064991 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3700722c0 0x7fc37006e510 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc368009e50 tx=0x7fc36800b330 comp rx=0 tx=0).stop 2023-11-24T06:08:36.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.544+0000 7fc37749f700 1 -- 172.21.15.91:0/1380064991 shutdown_connections 2023-11-24T06:08:36.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.544+0000 7fc37749f700 1 --2- 172.21.15.91:0/1380064991 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3700722c0 0x7fc37006e510 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.544+0000 7fc37749f700 1 --2- 172.21.15.91:0/1380064991 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc370143ae0 0x7fc3700b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.544+0000 7fc37749f700 1 -- 172.21.15.91:0/1380064991 >> 172.21.15.91:0/1380064991 conn(0x7fc3700b2740 msgr2=0x7fc3700b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:36.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.544+0000 7fc37749f700 1 -- 172.21.15.91:0/1380064991 shutdown_connections 2023-11-24T06:08:36.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.544+0000 7fc37749f700 1 -- 172.21.15.91:0/1380064991 wait complete. 2023-11-24T06:08:36.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.545+0000 7fc37749f700 1 Processor -- start 2023-11-24T06:08:36.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.546+0000 7fc37749f700 1 -- start start 2023-11-24T06:08:36.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.546+0000 7fc37749f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc3700722c0 0x7fc370154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.551 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.546+0000 7fc37749f700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc370143ae0 0x7fc370156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.546+0000 7fc37749f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3701440f0 0x7fc370157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.546+0000 7fc37749f700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3701f1eb0 con 0x7fc370143ae0 2023-11-24T06:08:36.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.546+0000 7fc37749f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3701e4220 con 0x7fc3701440f0 2023-11-24T06:08:36.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.546+0000 7fc37749f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc370143300 con 0x7fc3700722c0 2023-11-24T06:08:36.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.546+0000 7fc374a3a700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc370143ae0 0x7fc370156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.546+0000 7fc374a3a700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc370143ae0 0x7fc370156df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58918/0 (socket says 172.21.15.91:58918) 2023-11-24T06:08:36.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.546+0000 7fc374a3a700 1 -- 172.21.15.91:0/1376555707 learned_addr learned my addr 172.21.15.91:0/1376555707 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:36.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.547+0000 7fc375a3c700 1 --2- 172.21.15.91:0/1376555707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3701440f0 0x7fc370157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.547+0000 7fc37523b700 1 --2- 172.21.15.91:0/1376555707 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc3700722c0 0x7fc370154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.547+0000 7fc374a3a700 1 -- 172.21.15.91:0/1376555707 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc3700722c0 msgr2=0x7fc370154330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.547+0000 7fc374a3a700 1 --2- 172.21.15.91:0/1376555707 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc3700722c0 0x7fc370154330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.547+0000 7fc374a3a700 1 -- 172.21.15.91:0/1376555707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3701440f0 msgr2=0x7fc370157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.552 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.547+0000 7fc374a3a700 1 --2- 172.21.15.91:0/1376555707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3701440f0 0x7fc370157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.547+0000 7fc374a3a700 1 -- 172.21.15.91:0/1376555707 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c0094a0 con 0x7fc370143ae0 2023-11-24T06:08:36.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.547+0000 7fc374a3a700 1 --2- 172.21.15.91:0/1376555707 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc370143ae0 0x7fc370156df0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc36000c500 tx=0x7fc36000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:36.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.547+0000 7fc365ffb700 1 -- 172.21.15.91:0/1376555707 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc360005cb0 con 0x7fc370143ae0 2023-11-24T06:08:36.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.547+0000 7fc365ffb700 1 -- 172.21.15.91:0/1376555707 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc360005e10 con 0x7fc370143ae0 2023-11-24T06:08:36.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.547+0000 7fc37749f700 1 -- 172.21.15.91:0/1376555707 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc368009b30 con 0x7fc370143ae0 2023-11-24T06:08:36.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.547+0000 7fc365ffb700 1 -- 172.21.15.91:0/1376555707 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3600072d0 con 0x7fc370143ae0 2023-11-24T06:08:36.553 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.547+0000 7fc37749f700 1 -- 172.21.15.91:0/1376555707 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc370142a60 con 0x7fc370143ae0 2023-11-24T06:08:36.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.548+0000 7fc37749f700 1 -- 172.21.15.91:0/1376555707 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc370075480 con 0x7fc370143ae0 2023-11-24T06:08:36.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.549+0000 7fc365ffb700 1 -- 172.21.15.91:0/1376555707 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc360007430 con 0x7fc370143ae0 2023-11-24T06:08:36.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.549+0000 7fc365ffb700 1 --2- 172.21.15.91:0/1376555707 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc35406aed0 0x7fc35406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.549+0000 7fc365ffb700 1 -- 172.21.15.91:0/1376555707 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc360088790 con 0x7fc370143ae0 2023-11-24T06:08:36.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.549+0000 7fc37523b700 1 --2- 172.21.15.91:0/1376555707 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc35406aed0 0x7fc35406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.554 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.549+0000 7fc37523b700 1 --2- 172.21.15.91:0/1376555707 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc35406aed0 0x7fc35406d390 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009d70 tx=0x7fc36c009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:36.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.554+0000 7fc365ffb700 1 -- 172.21.15.91:0/1376555707 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3600538f0 con 0x7fc370143ae0 2023-11-24T06:08:36.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.743+0000 7fc37749f700 1 -- 172.21.15.91:0/1376555707 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc3700b5d20 con 0x7fc370143ae0 2023-11-24T06:08:36.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.744+0000 7fc365ffb700 1 -- 172.21.15.91:0/1376555707 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc3600170a0 con 0x7fc370143ae0 2023-11-24T06:08:36.747 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 64 2023-11-24T06:08:36.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.746+0000 7fc37749f700 1 -- 172.21.15.91:0/1376555707 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc35406aed0 msgr2=0x7fc35406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.746+0000 7fc37749f700 1 --2- 172.21.15.91:0/1376555707 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc35406aed0 0x7fc35406d390 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009d70 tx=0x7fc36c009210 comp rx=0 tx=0).stop 2023-11-24T06:08:36.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.746+0000 7fc37749f700 1 -- 172.21.15.91:0/1376555707 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc370143ae0 msgr2=0x7fc370156df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.746+0000 7fc37749f700 1 --2- 172.21.15.91:0/1376555707 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc370143ae0 0x7fc370156df0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc36000c500 tx=0x7fc36000c810 comp rx=0 tx=0).stop 2023-11-24T06:08:36.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.746+0000 7fc37749f700 1 -- 172.21.15.91:0/1376555707 shutdown_connections 2023-11-24T06:08:36.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.746+0000 7fc37749f700 1 --2- 172.21.15.91:0/1376555707 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc35406aed0 0x7fc35406d390 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.746+0000 7fc37749f700 1 --2- 172.21.15.91:0/1376555707 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc3701440f0 0x7fc370157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.746+0000 7fc37749f700 1 --2- 172.21.15.91:0/1376555707 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc3700722c0 0x7fc370154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.746+0000 7fc37749f700 1 --2- 172.21.15.91:0/1376555707 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc370143ae0 0x7fc370156df0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.746+0000 7fc37749f700 1 -- 172.21.15.91:0/1376555707 >> 172.21.15.91:0/1376555707 conn(0x7fc3700b2740 msgr2=0x7fc37015b530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:36.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.746+0000 7fc37749f700 1 -- 172.21.15.91:0/1376555707 shutdown_connections 2023-11-24T06:08:36.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.746+0000 7fc37749f700 1 -- 172.21.15.91:0/1376555707 wait complete. 2023-11-24T06:08:36.761 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T06:08:36.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.879+0000 7f4cb903c700 1 Processor -- start 2023-11-24T06:08:36.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.879+0000 7f4cb903c700 1 -- start start 2023-11-24T06:08:36.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.880+0000 7f4cb903c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4cb415e1c0 0x7f4cb406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.880+0000 7f4cb903c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4cb40b0e90 0x7f4cb406f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.880+0000 7f4cb903c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb40d1230 con 0x7f4cb406fa80 2023-11-24T06:08:36.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.880+0000 7f4cb903c700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb406ff10 con 0x7f4cb415e1c0 2023-11-24T06:08:36.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.880+0000 7f4cb903c700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb40d0c70 con 0x7f4cb40b0e90 2023-11-24T06:08:36.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.880+0000 7f4cb2d9d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4cb415e1c0 0x7f4cb406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.880+0000 7f4cb2d9d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4cb415e1c0 0x7f4cb406eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58922/0 (socket says 172.21.15.91:58922) 2023-11-24T06:08:36.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.880+0000 7f4cb2d9d700 1 -- 172.21.15.91:0/1575614114 learned_addr learned my addr 172.21.15.91:0/1575614114 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:36.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.880+0000 7f4cb2d9d700 1 -- 172.21.15.91:0/1575614114 >> v2:172.21.15.94:3300/0 conn(0x7f4cb40b0e90 msgr2=0x7f4cb406f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:36.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.880+0000 7f4cb2d9d700 1 --2- 172.21.15.91:0/1575614114 >> v2:172.21.15.94:3300/0 conn(0x7f4cb40b0e90 0x7f4cb406f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.880+0000 7f4cb2d9d700 1 -- 172.21.15.91:0/1575614114 >> v1:172.21.15.94:6789/0 conn(0x7f4cb406fa80 legacy=0x7f4cb4161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:36.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.880+0000 7f4cb2d9d700 1 -- 172.21.15.91:0/1575614114 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb40d0f50 con 0x7f4cb415e1c0 2023-11-24T06:08:36.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.880+0000 7f4cb2d9d700 1 --2- 172.21.15.91:0/1575614114 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4cb415e1c0 0x7f4cb406eef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4071ba0 tx=0x7f4ca8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7111df79d487c37e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:36.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.880+0000 7f4cb1d9b700 1 -- 172.21.15.91:0/1575614114 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8004230 con 0x7f4cb415e1c0 2023-11-24T06:08:36.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.880+0000 7f4cb1d9b700 1 -- 172.21.15.91:0/1575614114 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca8004390 con 0x7f4cb415e1c0 2023-11-24T06:08:36.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.880+0000 7f4cb1d9b700 1 -- 172.21.15.91:0/1575614114 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca80036a0 con 0x7f4cb415e1c0 2023-11-24T06:08:36.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.881+0000 7f4cb903c700 1 -- 172.21.15.91:0/1575614114 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4cb415e1c0 msgr2=0x7f4cb406eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.881+0000 7f4cb903c700 1 --2- 172.21.15.91:0/1575614114 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4cb415e1c0 0x7f4cb406eef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4071ba0 tx=0x7f4ca8009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.881+0000 7f4cb903c700 1 -- 172.21.15.91:0/1575614114 shutdown_connections 2023-11-24T06:08:36.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.881+0000 7f4cb903c700 1 --2- 172.21.15.91:0/1575614114 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4cb415e1c0 0x7f4cb406eef0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.881+0000 7f4cb903c700 1 --2- 172.21.15.91:0/1575614114 >> v2:172.21.15.94:3300/0 conn(0x7f4cb40b0e90 0x7f4cb406f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.881+0000 7f4cb903c700 1 -- 172.21.15.91:0/1575614114 >> 172.21.15.91:0/1575614114 conn(0x7f4cb40b3720 msgr2=0x7f4cb40b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:36.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.881+0000 7f4cb903c700 1 -- 172.21.15.91:0/1575614114 shutdown_connections 2023-11-24T06:08:36.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.881+0000 7f4cb903c700 1 -- 172.21.15.91:0/1575614114 wait complete. 2023-11-24T06:08:36.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.883+0000 7f4cb903c700 1 Processor -- start 2023-11-24T06:08:36.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.883+0000 7f4cb903c700 1 -- start start 2023-11-24T06:08:36.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.883+0000 7f4cb903c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4cb406fa80 0x7f4cb4147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.883+0000 7f4cb903c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cb40b0e90 0x7f4cb41586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.883+0000 7f4cb903c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4cb415e1c0 0x7f4cb4158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.883+0000 7f4cb903c700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb40b51d0 con 0x7f4cb406fa80 2023-11-24T06:08:36.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.883+0000 7f4cb903c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb4155da0 con 0x7f4cb40b0e90 2023-11-24T06:08:36.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.883+0000 7f4cb903c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb4156920 con 0x7f4cb415e1c0 2023-11-24T06:08:36.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.883+0000 7f4cb2d9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4cb406fa80 0x7f4cb4147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.883+0000 7f4cb2d9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4cb406fa80 0x7f4cb4147ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58928/0 (socket says 172.21.15.91:58928) 2023-11-24T06:08:36.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.883+0000 7f4cb2d9d700 1 -- 172.21.15.91:0/4192646722 learned_addr learned my addr 172.21.15.91:0/4192646722 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:36.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.884+0000 7f4cb2d9d700 1 -- 172.21.15.91:0/4192646722 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4cb415e1c0 msgr2=0x7f4cb4158bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.884+0000 7f4cb2d9d700 1 --2- 172.21.15.91:0/4192646722 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4cb415e1c0 0x7f4cb4158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.884+0000 7f4cb2d9d700 1 -- 172.21.15.91:0/4192646722 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cb40b0e90 msgr2=0x7f4cb41586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:36.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.884+0000 7f4cb259c700 1 --2- 172.21.15.91:0/4192646722 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cb40b0e90 0x7f4cb41586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.884+0000 7f4cb2d9d700 1 --2- 172.21.15.91:0/4192646722 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cb40b0e90 0x7f4cb41586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:36.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.884+0000 7f4cb2d9d700 1 -- 172.21.15.91:0/4192646722 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca8009550 con 0x7f4cb406fa80 2023-11-24T06:08:36.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.884+0000 7f4cb2d9d700 1 --2- 172.21.15.91:0/4192646722 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4cb406fa80 0x7f4cb4147ea0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4072340 tx=0x7f4ca8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:36.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.884+0000 7f4c9b7fe700 1 -- 172.21.15.91:0/4192646722 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8011630 con 0x7f4cb406fa80 2023-11-24T06:08:36.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.884+0000 7f4c9b7fe700 1 -- 172.21.15.91:0/4192646722 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca8011790 con 0x7f4cb406fa80 2023-11-24T06:08:36.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.884+0000 7f4c9b7fe700 1 -- 172.21.15.91:0/4192646722 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8011b40 con 0x7f4cb406fa80 2023-11-24T06:08:36.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.884+0000 7f4cb903c700 1 -- 172.21.15.91:0/4192646722 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb4155ff0 con 0x7f4cb406fa80 2023-11-24T06:08:36.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.884+0000 7f4cb903c700 1 -- 172.21.15.91:0/4192646722 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb41565b0 con 0x7f4cb406fa80 2023-11-24T06:08:36.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.885+0000 7f4cb903c700 1 -- 172.21.15.91:0/4192646722 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb4068670 con 0x7f4cb406fa80 2023-11-24T06:08:36.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.886+0000 7f4c9b7fe700 1 -- 172.21.15.91:0/4192646722 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4ca8011ca0 con 0x7f4cb406fa80 2023-11-24T06:08:36.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.886+0000 7f4c9b7fe700 1 --2- 172.21.15.91:0/4192646722 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4ca006aed0 0x7f4ca006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:36.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.886+0000 7f4cb259c700 1 --2- 172.21.15.91:0/4192646722 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4ca006aed0 0x7f4ca006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:36.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.886+0000 7f4c9b7fe700 1 -- 172.21.15.91:0/4192646722 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ca808ac30 con 0x7f4cb406fa80 2023-11-24T06:08:36.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.887+0000 7f4cb259c700 1 --2- 172.21.15.91:0/4192646722 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4ca006aed0 0x7f4ca006d390 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4006fd0 tx=0x7f4ca4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:36.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:36.891+0000 7f4c9b7fe700 1 -- 172.21.15.91:0/4192646722 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4ca8055d70 con 0x7f4cb406fa80 2023-11-24T06:08:37.101 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.098+0000 7f4cb903c700 1 -- 172.21.15.91:0/4192646722 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4cb40b4240 con 0x7f4cb406fa80 2023-11-24T06:08:37.101 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.099+0000 7f4c9b7fe700 1 -- 172.21.15.91:0/4192646722 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4ca8059390 con 0x7f4cb406fa80 2023-11-24T06:08:37.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.101+0000 7f4cb903c700 1 -- 172.21.15.91:0/4192646722 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4ca006aed0 msgr2=0x7f4ca006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:37.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.101+0000 7f4cb903c700 1 --2- 172.21.15.91:0/4192646722 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4ca006aed0 0x7f4ca006d390 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4006fd0 tx=0x7f4ca4008040 comp rx=0 tx=0).stop 2023-11-24T06:08:37.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.101+0000 7f4cb903c700 1 -- 172.21.15.91:0/4192646722 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4cb406fa80 msgr2=0x7f4cb4147ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:37.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.101+0000 7f4cb903c700 1 --2- 172.21.15.91:0/4192646722 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4cb406fa80 0x7f4cb4147ea0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4072340 tx=0x7f4ca8004750 comp rx=0 tx=0).stop 2023-11-24T06:08:37.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.101+0000 7f4cb903c700 1 -- 172.21.15.91:0/4192646722 shutdown_connections 2023-11-24T06:08:37.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.101+0000 7f4cb903c700 1 --2- 172.21.15.91:0/4192646722 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4ca006aed0 0x7f4ca006d390 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.101+0000 7f4cb903c700 1 --2- 172.21.15.91:0/4192646722 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4cb415e1c0 0x7f4cb4158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.101+0000 7f4cb903c700 1 --2- 172.21.15.91:0/4192646722 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4cb406fa80 0x7f4cb4147ea0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.101+0000 7f4cb903c700 1 --2- 172.21.15.91:0/4192646722 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4cb40b0e90 0x7f4cb41586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.101+0000 7f4cb903c700 1 -- 172.21.15.91:0/4192646722 >> 172.21.15.91:0/4192646722 conn(0x7f4cb40b3720 msgr2=0x7f4cb406db70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:37.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.101+0000 7f4cb903c700 1 -- 172.21.15.91:0/4192646722 shutdown_connections 2023-11-24T06:08:37.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.101+0000 7f4cb903c700 1 -- 172.21.15.91:0/4192646722 wait complete. 2023-11-24T06:08:37.105 INFO:teuthology.orchestra.run.smithi091.stderr:listed 0 entries 2023-11-24T06:08:37.115 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T06:08:37.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:37.116 DEBUG:teuthology.orchestra.run.smithi091:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-11-24T06:08:37.190 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi091.front.sepia.ceph.com 2023-11-24T06:08:37.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:37.190 DEBUG:teuthology.orchestra.run.smithi091:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T06:08:37.225 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:08:37.225 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:08:37.246 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:08:37.246 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:08:37.308 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:08:37.308 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:08:37.368 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:37.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.486+0000 7f4c3121f700 1 Processor -- start 2023-11-24T06:08:37.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.486+0000 7f4c3121f700 1 -- start start 2023-11-24T06:08:37.491 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.487+0000 7f4c3121f700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f4c2c0722c0 0x7f4c2c0b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:37.491 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.487+0000 7f4c3121f700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4c2c1440f0 0x7f4c2c06e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:37.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.487+0000 7f4c3121f700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c143ec0 con 0x7f4c2c143ae0 2023-11-24T06:08:37.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.487+0000 7f4c3121f700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2c0b40a0 con 0x7f4c2c1440f0 2023-11-24T06:08:37.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.487+0000 7f4c3121f700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2c0b3ae0 con 0x7f4c2c0722c0 2023-11-24T06:08:37.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.487+0000 7f4c2ad9d700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f4c2c143ae0 0x7f4c2c071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53870/0 (socket says 172.21.15.91:53870) 2023-11-24T06:08:37.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.487+0000 7f4c2b59e700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4c2c1440f0 0x7f4c2c06e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:37.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.487+0000 7f4c2ad9d700 1 -- 172.21.15.91:0/3607027963 learned_addr learned my addr 172.21.15.91:0/3607027963 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:37.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.488+0000 7f4c2b59e700 1 -- 172.21.15.91:0/3607027963 >> v2:172.21.15.94:3300/0 conn(0x7f4c2c0722c0 msgr2=0x7f4c2c0b51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:37.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.488+0000 7f4c2b59e700 1 --2- 172.21.15.91:0/3607027963 >> v2:172.21.15.94:3300/0 conn(0x7f4c2c0722c0 0x7f4c2c0b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.488+0000 7f4c2b59e700 1 -- 172.21.15.91:0/3607027963 >> v1:172.21.15.91:6789/0 conn(0x7f4c2c143ae0 legacy=0x7f4c2c071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:37.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.488+0000 7f4c29d9b700 1 -- 172.21.15.91:0/3607027963 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618976235 0 0) 0x7f4c2c143ec0 con 0x7f4c2c143ae0 2023-11-24T06:08:37.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.488+0000 7f4c2b59e700 1 -- 172.21.15.91:0/3607027963 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c0b3dc0 con 0x7f4c2c1440f0 2023-11-24T06:08:37.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.488+0000 7f4c2b59e700 1 --2- 172.21.15.91:0/3607027963 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c2c1440f0 0x7f4c2c06e510 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0d1b00 tx=0x7f4c1c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a75a1d437f8d28 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:37.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.488+0000 7f4c29d9b700 1 -- 172.21.15.91:0/3607027963 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c011830 con 0x7f4c2c1440f0 2023-11-24T06:08:37.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.488+0000 7f4c29d9b700 1 -- 172.21.15.91:0/3607027963 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1c011990 con 0x7f4c2c1440f0 2023-11-24T06:08:37.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.488+0000 7f4c29d9b700 1 -- 172.21.15.91:0/3607027963 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c011d20 con 0x7f4c2c1440f0 2023-11-24T06:08:37.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.488+0000 7f4c3121f700 1 -- 172.21.15.91:0/3607027963 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c2c1440f0 msgr2=0x7f4c2c06e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:37.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.488+0000 7f4c3121f700 1 --2- 172.21.15.91:0/3607027963 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c2c1440f0 0x7f4c2c06e510 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0d1b00 tx=0x7f4c1c009b20 comp rx=0 tx=0).stop 2023-11-24T06:08:37.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.488+0000 7f4c3121f700 1 -- 172.21.15.91:0/3607027963 shutdown_connections 2023-11-24T06:08:37.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.488+0000 7f4c3121f700 1 --2- 172.21.15.91:0/3607027963 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c2c1440f0 0x7f4c2c06e510 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.488+0000 7f4c3121f700 1 --2- 172.21.15.91:0/3607027963 >> v2:172.21.15.94:3300/0 conn(0x7f4c2c0722c0 0x7f4c2c0b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.488+0000 7f4c3121f700 1 -- 172.21.15.91:0/3607027963 >> 172.21.15.91:0/3607027963 conn(0x7f4c2c0b2740 msgr2=0x7f4c2c0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:37.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.489+0000 7f4c3121f700 1 -- 172.21.15.91:0/3607027963 shutdown_connections 2023-11-24T06:08:37.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.489+0000 7f4c3121f700 1 -- 172.21.15.91:0/3607027963 wait complete. 2023-11-24T06:08:37.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.490+0000 7f4c3121f700 1 Processor -- start 2023-11-24T06:08:37.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.490+0000 7f4c3121f700 1 -- start start 2023-11-24T06:08:37.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c3121f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c2c0722c0 0x7f4c2c13d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:37.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c3121f700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c2c143ae0 0x7f4c2c13c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:37.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c3121f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4c2c1440f0 0x7f4c2c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:37.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c3121f700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c13dbf0 con 0x7f4c2c143ae0 2023-11-24T06:08:37.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c3121f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c13ded0 con 0x7f4c2c0722c0 2023-11-24T06:08:37.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c3121f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c13e1b0 con 0x7f4c2c1440f0 2023-11-24T06:08:37.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c2a59c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c2c143ae0 0x7f4c2c13c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:37.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c2a59c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c2c143ae0 0x7f4c2c13c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58944/0 (socket says 172.21.15.91:58944) 2023-11-24T06:08:37.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c2a59c700 1 -- 172.21.15.91:0/3897746305 learned_addr learned my addr 172.21.15.91:0/3897746305 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:37.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c2b59e700 1 --2- 172.21.15.91:0/3897746305 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4c2c1440f0 0x7f4c2c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:37.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c2ad9d700 1 --2- 172.21.15.91:0/3897746305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c2c0722c0 0x7f4c2c13d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:37.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c2a59c700 1 -- 172.21.15.91:0/3897746305 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4c2c1440f0 msgr2=0x7f4c2c06ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:37.498 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c2a59c700 1 --2- 172.21.15.91:0/3897746305 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4c2c1440f0 0x7f4c2c06ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.498 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c2a59c700 1 -- 172.21.15.91:0/3897746305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c2c0722c0 msgr2=0x7f4c2c13d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:37.498 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c2a59c700 1 --2- 172.21.15.91:0/3897746305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c2c0722c0 0x7f4c2c13d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.498 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c2a59c700 1 -- 172.21.15.91:0/3897746305 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1c0094a0 con 0x7f4c2c143ae0 2023-11-24T06:08:37.498 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c2b59e700 1 --2- 172.21.15.91:0/3897746305 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4c2c1440f0 0x7f4c2c06ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:37.498 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c2a59c700 1 --2- 172.21.15.91:0/3897746305 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c2c143ae0 0x7f4c2c13c420 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000b530 tx=0x7f4c2000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:37.498 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c137fe700 1 -- 172.21.15.91:0/3897746305 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20011830 con 0x7f4c2c143ae0 2023-11-24T06:08:37.498 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c137fe700 1 -- 172.21.15.91:0/3897746305 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c200119b0 con 0x7f4c2c143ae0 2023-11-24T06:08:37.498 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c3121f700 1 -- 172.21.15.91:0/3897746305 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2c13e490 con 0x7f4c2c143ae0 2023-11-24T06:08:37.498 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.491+0000 7f4c137fe700 1 -- 172.21.15.91:0/3897746305 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20011d60 con 0x7f4c2c143ae0 2023-11-24T06:08:37.498 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.492+0000 7f4c3121f700 1 -- 172.21.15.91:0/3897746305 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2c13f970 con 0x7f4c2c143ae0 2023-11-24T06:08:37.499 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.493+0000 7f4c3121f700 1 -- 172.21.15.91:0/3897746305 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2c068670 con 0x7f4c2c143ae0 2023-11-24T06:08:37.499 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.493+0000 7f4c137fe700 1 -- 172.21.15.91:0/3897746305 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4c2001a400 con 0x7f4c2c143ae0 2023-11-24T06:08:37.499 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.493+0000 7f4c137fe700 1 --2- 172.21.15.91:0/3897746305 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c1806aed0 0x7f4c1806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:37.499 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.493+0000 7f4c137fe700 1 -- 172.21.15.91:0/3897746305 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c200897c0 con 0x7f4c2c143ae0 2023-11-24T06:08:37.499 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.493+0000 7f4c2ad9d700 1 --2- 172.21.15.91:0/3897746305 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c1806aed0 0x7f4c1806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:37.499 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.494+0000 7f4c2ad9d700 1 --2- 172.21.15.91:0/3897746305 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c1806aed0 0x7f4c1806d390 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4c240012e0 tx=0x7f4c24001970 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:37.500 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.498+0000 7f4c137fe700 1 -- 172.21.15.91:0/3897746305 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c20054760 con 0x7f4c2c143ae0 2023-11-24T06:08:37.691 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.688+0000 7f4c3121f700 1 -- 172.21.15.91:0/3897746305 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c2c0b5d20 con 0x7f4c2c143ae0 2023-11-24T06:08:37.692 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.689+0000 7f4c137fe700 1 -- 172.21.15.91:0/3897746305 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7f4c20015090 con 0x7f4c2c143ae0 2023-11-24T06:08:37.693 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:37.693 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":20,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:02.415612+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:37.694 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.691+0000 7f4c3121f700 1 -- 172.21.15.91:0/3897746305 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c1806aed0 msgr2=0x7f4c1806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:37.695 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.691+0000 7f4c3121f700 1 --2- 172.21.15.91:0/3897746305 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c1806aed0 0x7f4c1806d390 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4c240012e0 tx=0x7f4c24001970 comp rx=0 tx=0).stop 2023-11-24T06:08:37.695 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.691+0000 7f4c3121f700 1 -- 172.21.15.91:0/3897746305 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c2c143ae0 msgr2=0x7f4c2c13c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:37.695 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.691+0000 7f4c3121f700 1 --2- 172.21.15.91:0/3897746305 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c2c143ae0 0x7f4c2c13c420 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000b530 tx=0x7f4c2000b8f0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.695 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.691+0000 7f4c3121f700 1 -- 172.21.15.91:0/3897746305 shutdown_connections 2023-11-24T06:08:37.695 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.691+0000 7f4c3121f700 1 --2- 172.21.15.91:0/3897746305 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c1806aed0 0x7f4c1806d390 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.695 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.691+0000 7f4c3121f700 1 --2- 172.21.15.91:0/3897746305 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c2c0722c0 0x7f4c2c13d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.695 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.691+0000 7f4c3121f700 1 --2- 172.21.15.91:0/3897746305 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4c2c1440f0 0x7f4c2c06ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.695 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.691+0000 7f4c3121f700 1 --2- 172.21.15.91:0/3897746305 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c2c143ae0 0x7f4c2c13c420 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.695 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.691+0000 7f4c3121f700 1 -- 172.21.15.91:0/3897746305 >> 172.21.15.91:0/3897746305 conn(0x7f4c2c0b2740 msgr2=0x7f4c2c1410d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:37.695 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.691+0000 7f4c3121f700 1 -- 172.21.15.91:0/3897746305 shutdown_connections 2023-11-24T06:08:37.695 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.691+0000 7f4c3121f700 1 -- 172.21.15.91:0/3897746305 wait complete. 2023-11-24T06:08:37.706 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:07:58.710106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:07:59.774819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:08:37.707 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:08:37.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.830+0000 7fe0fc418700 1 Processor -- start 2023-11-24T06:08:37.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.830+0000 7fe0fc418700 1 -- start start 2023-11-24T06:08:37.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.831+0000 7fe0fc418700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe0f40722c0 0x7fe0f40b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:37.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.831+0000 7fe0fc418700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe0f41440f0 0x7fe0f406e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:37.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.831+0000 7fe0fc418700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f4143ec0 con 0x7fe0f4143ae0 2023-11-24T06:08:37.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.831+0000 7fe0fc418700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0f40b40a0 con 0x7fe0f40722c0 2023-11-24T06:08:37.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.831+0000 7fe0fc418700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f40b3ae0 con 0x7fe0f41440f0 2023-11-24T06:08:37.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.831+0000 7fe0fa1b4700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fe0f4143ae0 0x7fe0f4071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53884/0 (socket says 172.21.15.91:53884) 2023-11-24T06:08:37.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.831+0000 7fe0fa1b4700 1 -- 172.21.15.91:0/3893090944 learned_addr learned my addr 172.21.15.91:0/3893090944 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:37.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.831+0000 7fe0f91b2700 1 -- 172.21.15.91:0/3893090944 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869497663 0 0) 0x7fe0f4143ec0 con 0x7fe0f4143ae0 2023-11-24T06:08:37.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.831+0000 7fe0f91b2700 1 -- 172.21.15.91:0/3893090944 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e00035b0 con 0x7fe0f4143ae0 2023-11-24T06:08:37.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.831+0000 7fe0fa9b5700 1 --2- 172.21.15.91:0/3893090944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe0f41440f0 0x7fe0f406e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:37.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.831+0000 7fe0f99b3700 1 --2- 172.21.15.91:0/3893090944 >> v2:172.21.15.94:3300/0 conn(0x7fe0f40722c0 0x7fe0f40b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:37.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f91b2700 1 -- 172.21.15.91:0/3893090944 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3919251325 0 0) 0x7fe0e00035b0 con 0x7fe0f4143ae0 2023-11-24T06:08:37.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f91b2700 1 -- 172.21.15.91:0/3893090944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe0f41440f0 msgr2=0x7fe0f406e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:37.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f91b2700 1 --2- 172.21.15.91:0/3893090944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe0f41440f0 0x7fe0f406e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f91b2700 1 -- 172.21.15.91:0/3893090944 >> v2:172.21.15.94:3300/0 conn(0x7fe0f40722c0 msgr2=0x7fe0f40b51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:37.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f91b2700 1 --2- 172.21.15.91:0/3893090944 >> v2:172.21.15.94:3300/0 conn(0x7fe0f40722c0 0x7fe0f40b51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f91b2700 1 -- 172.21.15.91:0/3893090944 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f40b3dc0 con 0x7fe0f4143ae0 2023-11-24T06:08:37.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f91b2700 1 -- 172.21.15.91:0/3893090944 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7fe0e4001670 con 0x7fe0f4143ae0 2023-11-24T06:08:37.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f91b2700 1 -- 172.21.15.91:0/3893090944 >> v1:172.21.15.91:6789/0 conn(0x7fe0f4143ae0 legacy=0x7fe0f4071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:37.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f91b2700 1 --2- 172.21.15.91:0/3893090944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe0e0003a00 0x7fe0e0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:37.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f91b2700 1 --2- 172.21.15.91:0/3893090944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe0f41440f0 0x7fe0e0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:37.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f91b2700 1 --2- 172.21.15.91:0/3893090944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe0e0004d90 0x7fe0e0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:37.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f91b2700 1 -- 172.21.15.91:0/3893090944 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f40b40a0 con 0x7fe0e0004d90 2023-11-24T06:08:37.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f91b2700 1 -- 172.21.15.91:0/3893090944 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f40b3ae0 con 0x7fe0e0003a00 2023-11-24T06:08:37.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f91b2700 1 -- 172.21.15.91:0/3893090944 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e000ba10 con 0x7fe0f41440f0 2023-11-24T06:08:37.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0fa9b5700 1 --2- 172.21.15.91:0/3893090944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe0e0004d90 0x7fe0e0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:37.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0fa1b4700 1 --2- 172.21.15.91:0/3893090944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe0e0003a00 0x7fe0e0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:37.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f99b3700 1 --2- 172.21.15.91:0/3893090944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe0f41440f0 0x7fe0e0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:37.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0fa9b5700 1 -- 172.21.15.91:0/3893090944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe0f41440f0 msgr2=0x7fe0e0004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:37.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0fa9b5700 1 --2- 172.21.15.91:0/3893090944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe0f41440f0 0x7fe0e0004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0fa9b5700 1 -- 172.21.15.91:0/3893090944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe0e0003a00 msgr2=0x7fe0e0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:37.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0fa9b5700 1 --2- 172.21.15.91:0/3893090944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe0e0003a00 0x7fe0e0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0fa9b5700 1 -- 172.21.15.91:0/3893090944 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e00035b0 con 0x7fe0e0004d90 2023-11-24T06:08:37.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0fa1b4700 1 --2- 172.21.15.91:0/3893090944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe0e0003a00 0x7fe0e0003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:37.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0fa9b5700 1 --2- 172.21.15.91:0/3893090944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe0e0004d90 0x7fe0e0005210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009fc0 tx=0x7fe0e0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c72620b493c52e71 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:37.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f91b2700 1 -- 172.21.15.91:0/3893090944 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec00ec30 con 0x7fe0e0004d90 2023-11-24T06:08:37.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.832+0000 7fe0f91b2700 1 -- 172.21.15.91:0/3893090944 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0ec00f040 con 0x7fe0e0004d90 2023-11-24T06:08:37.839 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.833+0000 7fe0f91b2700 1 -- 172.21.15.91:0/3893090944 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec014480 con 0x7fe0e0004d90 2023-11-24T06:08:37.840 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.833+0000 7fe0fc418700 1 -- 172.21.15.91:0/3893090944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe0e0004d90 msgr2=0x7fe0e0005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:37.840 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.833+0000 7fe0fc418700 1 --2- 172.21.15.91:0/3893090944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe0e0004d90 0x7fe0e0005210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009fc0 tx=0x7fe0e0005880 comp rx=0 tx=0).stop 2023-11-24T06:08:37.840 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.833+0000 7fe0fa9b5700 1 -- 172.21.15.91:0/3893090944 reap_dead start 2023-11-24T06:08:37.840 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.833+0000 7fe0fc418700 1 -- 172.21.15.91:0/3893090944 shutdown_connections 2023-11-24T06:08:37.840 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.833+0000 7fe0fc418700 1 -- 172.21.15.91:0/3893090944 >> 172.21.15.91:0/3893090944 conn(0x7fe0f40b2740 msgr2=0x7fe0f40b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:37.840 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.833+0000 7fe0fc418700 1 -- 172.21.15.91:0/3893090944 shutdown_connections 2023-11-24T06:08:37.840 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.833+0000 7fe0fc418700 1 -- 172.21.15.91:0/3893090944 wait complete. 2023-11-24T06:08:37.843 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.834+0000 7fe0fc418700 1 Processor -- start 2023-11-24T06:08:37.843 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.834+0000 7fe0fc418700 1 -- start start 2023-11-24T06:08:37.843 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0fc418700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe0e0004d90 0x7fe0f41545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:37.843 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0fc418700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe0f41f4ea0 0x7fe0f41f5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:37.843 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0fc418700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe0f41562a0 0x7fe0f4156720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:37.843 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0fc418700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f41f5860 con 0x7fe0e0004d90 2023-11-24T06:08:37.843 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0fc418700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f4157ca0 con 0x7fe0f41562a0 2023-11-24T06:08:37.843 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0fc418700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f4154af0 con 0x7fe0f41f4ea0 2023-11-24T06:08:37.843 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0fa1b4700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe0e0004d90 0x7fe0f41545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:37.843 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0fa1b4700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe0e0004d90 0x7fe0f41545b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58962/0 (socket says 172.21.15.91:58962) 2023-11-24T06:08:37.844 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0fa1b4700 1 -- 172.21.15.91:0/1592587253 learned_addr learned my addr 172.21.15.91:0/1592587253 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:37.844 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0fa9b5700 1 --2- 172.21.15.91:0/1592587253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe0f41562a0 0x7fe0f4156720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:37.844 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0f99b3700 1 --2- 172.21.15.91:0/1592587253 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe0f41f4ea0 0x7fe0f41f5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:37.844 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0fa1b4700 1 -- 172.21.15.91:0/1592587253 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe0f41f4ea0 msgr2=0x7fe0f41f5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:37.844 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0fa1b4700 1 --2- 172.21.15.91:0/1592587253 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe0f41f4ea0 0x7fe0f41f5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.844 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0fa1b4700 1 -- 172.21.15.91:0/1592587253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe0f41562a0 msgr2=0x7fe0f4156720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:37.844 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0fa1b4700 1 --2- 172.21.15.91:0/1592587253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe0f41562a0 0x7fe0f4156720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:37.844 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0fa1b4700 1 -- 172.21.15.91:0/1592587253 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f4157090 con 0x7fe0e0004d90 2023-11-24T06:08:37.844 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.835+0000 7fe0f99b3700 1 --2- 172.21.15.91:0/1592587253 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe0f41f4ea0 0x7fe0f41f5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:37.844 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.836+0000 7fe0fa1b4700 1 --2- 172.21.15.91:0/1592587253 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe0e0004d90 0x7fe0f41545b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe0e400e820 tx=0x7fe0e400ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:37.844 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.836+0000 7fe0eaffd700 1 -- 172.21.15.91:0/1592587253 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e400c620 con 0x7fe0e0004d90 2023-11-24T06:08:37.844 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.836+0000 7fe0eaffd700 1 -- 172.21.15.91:0/1592587253 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e400c780 con 0x7fe0e0004d90 2023-11-24T06:08:37.844 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.836+0000 7fe0eaffd700 1 -- 172.21.15.91:0/1592587253 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e400cb10 con 0x7fe0e0004d90 2023-11-24T06:08:37.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.836+0000 7fe0fc418700 1 -- 172.21.15.91:0/1592587253 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f4157370 con 0x7fe0e0004d90 2023-11-24T06:08:37.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.836+0000 7fe0fc418700 1 -- 172.21.15.91:0/1592587253 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f4157930 con 0x7fe0e0004d90 2023-11-24T06:08:37.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.837+0000 7fe0fc418700 1 -- 172.21.15.91:0/1592587253 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f4067a00 con 0x7fe0e0004d90 2023-11-24T06:08:37.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.837+0000 7fe0eaffd700 1 -- 172.21.15.91:0/1592587253 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe0e400cc70 con 0x7fe0e0004d90 2023-11-24T06:08:37.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.837+0000 7fe0eaffd700 1 --2- 172.21.15.91:0/1592587253 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe0d806aed0 0x7fe0d806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:37.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.838+0000 7fe0eaffd700 1 -- 172.21.15.91:0/1592587253 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0e40893f0 con 0x7fe0e0004d90 2023-11-24T06:08:37.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.838+0000 7fe0f99b3700 1 --2- 172.21.15.91:0/1592587253 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe0d806aed0 0x7fe0d806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:37.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.838+0000 7fe0f99b3700 1 --2- 172.21.15.91:0/1592587253 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe0d806aed0 0x7fe0d806d390 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4151010 tx=0x7fe0f0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:37.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:37.842+0000 7fe0eaffd700 1 -- 172.21.15.91:0/1592587253 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0e40544a0 con 0x7fe0e0004d90 2023-11-24T06:08:38.035 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.032+0000 7fe0fc418700 1 -- 172.21.15.91:0/1592587253 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe0f4157f80 con 0x7fe0e0004d90 2023-11-24T06:08:38.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.033+0000 7fe0eaffd700 1 -- 172.21.15.91:0/1592587253 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe0e4057ac0 con 0x7fe0e0004d90 2023-11-24T06:08:38.036 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:08:38.038 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.035+0000 7fe0fc418700 1 -- 172.21.15.91:0/1592587253 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe0d806aed0 msgr2=0x7fe0d806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.038 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.035+0000 7fe0fc418700 1 --2- 172.21.15.91:0/1592587253 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe0d806aed0 0x7fe0d806d390 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4151010 tx=0x7fe0f0008040 comp rx=0 tx=0).stop 2023-11-24T06:08:38.038 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.035+0000 7fe0fc418700 1 -- 172.21.15.91:0/1592587253 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe0e0004d90 msgr2=0x7fe0f41545b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.038 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.035+0000 7fe0fc418700 1 --2- 172.21.15.91:0/1592587253 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe0e0004d90 0x7fe0f41545b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe0e400e820 tx=0x7fe0e400ebe0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.038 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.035+0000 7fe0fc418700 1 -- 172.21.15.91:0/1592587253 shutdown_connections 2023-11-24T06:08:38.038 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.035+0000 7fe0fc418700 1 --2- 172.21.15.91:0/1592587253 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe0d806aed0 0x7fe0d806d390 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.038 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.035+0000 7fe0fc418700 1 --2- 172.21.15.91:0/1592587253 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe0f41562a0 0x7fe0f4156720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.038 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.035+0000 7fe0fc418700 1 --2- 172.21.15.91:0/1592587253 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe0e0004d90 0x7fe0f41545b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.035+0000 7fe0fc418700 1 --2- 172.21.15.91:0/1592587253 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe0f41f4ea0 0x7fe0f41f5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.035+0000 7fe0fc418700 1 -- 172.21.15.91:0/1592587253 >> 172.21.15.91:0/1592587253 conn(0x7fe0f40b2740 msgr2=0x7fe0f415e840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:38.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.035+0000 7fe0fc418700 1 -- 172.21.15.91:0/1592587253 shutdown_connections 2023-11-24T06:08:38.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.035+0000 7fe0fc418700 1 -- 172.21.15.91:0/1592587253 wait complete. 2023-11-24T06:08:38.049 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:08:38.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.170+0000 7f959187e700 1 Processor -- start 2023-11-24T06:08:38.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.170+0000 7f959187e700 1 -- start start 2023-11-24T06:08:38.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.171+0000 7f959187e700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f958c15e080 0x7f958c0cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.171+0000 7f959187e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f958c0d2440 0x7f958c0d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.171+0000 7f959187e700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f958c0d2de0 con 0x7f958c0d0740 2023-11-24T06:08:38.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.171+0000 7f959187e700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f958c0d14b0 con 0x7f958c15e080 2023-11-24T06:08:38.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.171+0000 7f959187e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f958c0ce9a0 con 0x7f958c0d2440 2023-11-24T06:08:38.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.171+0000 7f958affd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f958c15e080 0x7f958c0cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.171+0000 7f958affd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f958c15e080 0x7f958c0cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58970/0 (socket says 172.21.15.91:58970) 2023-11-24T06:08:38.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.171+0000 7f958b7fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f958c0d0740 0x7f958c0d0bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53908/0 (socket says 172.21.15.91:53908) 2023-11-24T06:08:38.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.171+0000 7f958affd700 1 -- 172.21.15.91:0/3181716733 learned_addr learned my addr 172.21.15.91:0/3181716733 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:38.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.171+0000 7f958a7fc700 1 --2- 172.21.15.91:0/3181716733 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f958c0d2440 0x7f958c0d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.171+0000 7f958affd700 1 -- 172.21.15.91:0/3181716733 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f958c0d2440 msgr2=0x7f958c0d28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:38.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.171+0000 7f958affd700 1 --2- 172.21.15.91:0/3181716733 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f958c0d2440 0x7f958c0d28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.171+0000 7f958affd700 1 -- 172.21.15.91:0/3181716733 >> v1:172.21.15.91:6789/0 conn(0x7f958c0d0740 legacy=0x7f958c0d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.171+0000 7f958affd700 1 -- 172.21.15.91:0/3181716733 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958c0cec40 con 0x7f958c15e080 2023-11-24T06:08:38.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.171+0000 7f9589ffb700 1 -- 172.21.15.91:0/3181716733 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510862940 0 0) 0x7f958c0d2de0 con 0x7f958c0d0740 2023-11-24T06:08:38.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.171+0000 7f958affd700 1 --2- 172.21.15.91:0/3181716733 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f958c15e080 0x7f958c0cfa00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f95800098f0 tx=0x7f9580009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58921347da18cc43 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:38.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.172+0000 7f9589ffb700 1 -- 172.21.15.91:0/3181716733 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000b8a0 con 0x7f958c15e080 2023-11-24T06:08:38.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.172+0000 7f9589ffb700 1 -- 172.21.15.91:0/3181716733 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958000ba00 con 0x7f958c15e080 2023-11-24T06:08:38.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.172+0000 7f9589ffb700 1 -- 172.21.15.91:0/3181716733 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000bd90 con 0x7f958c15e080 2023-11-24T06:08:38.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.172+0000 7f959187e700 1 -- 172.21.15.91:0/3181716733 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f958c15e080 msgr2=0x7f958c0cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.172+0000 7f959187e700 1 --2- 172.21.15.91:0/3181716733 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f958c15e080 0x7f958c0cfa00 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f95800098f0 tx=0x7f9580009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:38.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.172+0000 7f959187e700 1 -- 172.21.15.91:0/3181716733 shutdown_connections 2023-11-24T06:08:38.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.172+0000 7f959187e700 1 --2- 172.21.15.91:0/3181716733 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f958c15e080 0x7f958c0cfa00 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.172+0000 7f959187e700 1 --2- 172.21.15.91:0/3181716733 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f958c0d2440 0x7f958c0d28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.172+0000 7f959187e700 1 -- 172.21.15.91:0/3181716733 >> 172.21.15.91:0/3181716733 conn(0x7f958c09d180 msgr2=0x7f958c0d3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:38.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.172+0000 7f959187e700 1 -- 172.21.15.91:0/3181716733 shutdown_connections 2023-11-24T06:08:38.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.172+0000 7f959187e700 1 -- 172.21.15.91:0/3181716733 wait complete. 2023-11-24T06:08:38.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.173+0000 7f959187e700 1 Processor -- start 2023-11-24T06:08:38.181 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.173+0000 7f959187e700 1 -- start start 2023-11-24T06:08:38.181 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.174+0000 7f959187e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f958c0d0740 0x7f958c0cc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.181 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.174+0000 7f959187e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f958c0d2440 0x7f958c143d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.181 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.174+0000 7f959187e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f958c15e080 0x7f958c0ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.181 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.174+0000 7f959187e700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c0cca50 con 0x7f958c0d2440 2023-11-24T06:08:38.181 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.174+0000 7f959187e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c0c79c0 con 0x7f958c15e080 2023-11-24T06:08:38.181 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.174+0000 7f959187e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f958c0cab20 con 0x7f958c0d0740 2023-11-24T06:08:38.181 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.174+0000 7f958a7fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f958c0d2440 0x7f958c143d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.181 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.174+0000 7f958a7fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f958c0d2440 0x7f958c143d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58978/0 (socket says 172.21.15.91:58978) 2023-11-24T06:08:38.181 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.174+0000 7f958a7fc700 1 -- 172.21.15.91:0/4042758772 learned_addr learned my addr 172.21.15.91:0/4042758772 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:38.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.174+0000 7f958affd700 1 --2- 172.21.15.91:0/4042758772 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f958c0d0740 0x7f958c0cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.174+0000 7f958b7fe700 1 --2- 172.21.15.91:0/4042758772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f958c15e080 0x7f958c0ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.175+0000 7f958a7fc700 1 -- 172.21.15.91:0/4042758772 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f958c0d0740 msgr2=0x7f958c0cc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.175+0000 7f958a7fc700 1 --2- 172.21.15.91:0/4042758772 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f958c0d0740 0x7f958c0cc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.175+0000 7f958a7fc700 1 -- 172.21.15.91:0/4042758772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f958c15e080 msgr2=0x7f958c0ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.175+0000 7f958a7fc700 1 --2- 172.21.15.91:0/4042758772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f958c15e080 0x7f958c0ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.175+0000 7f958a7fc700 1 -- 172.21.15.91:0/4042758772 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9580009550 con 0x7f958c0d2440 2023-11-24T06:08:38.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.175+0000 7f958a7fc700 1 --2- 172.21.15.91:0/4042758772 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f958c0d2440 0x7f958c143d90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f957c00e810 tx=0x7f957c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:38.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.175+0000 7f9573fff700 1 -- 172.21.15.91:0/4042758772 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c00c070 con 0x7f958c0d2440 2023-11-24T06:08:38.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.175+0000 7f9573fff700 1 -- 172.21.15.91:0/4042758772 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957c013610 con 0x7f958c0d2440 2023-11-24T06:08:38.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.175+0000 7f959187e700 1 -- 172.21.15.91:0/4042758772 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958c0c7c80 con 0x7f958c0d2440 2023-11-24T06:08:38.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.175+0000 7f9573fff700 1 -- 172.21.15.91:0/4042758772 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957c0139c0 con 0x7f958c0d2440 2023-11-24T06:08:38.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.175+0000 7f959187e700 1 -- 172.21.15.91:0/4042758772 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958c0cc760 con 0x7f958c0d2440 2023-11-24T06:08:38.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.176+0000 7f959187e700 1 -- 172.21.15.91:0/4042758772 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f958c0c81a0 con 0x7f958c0d2440 2023-11-24T06:08:38.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.176+0000 7f9573fff700 1 -- 172.21.15.91:0/4042758772 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f957c013b20 con 0x7f958c0d2440 2023-11-24T06:08:38.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.177+0000 7f9573fff700 1 --2- 172.21.15.91:0/4042758772 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f957806aed0 0x7f957806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.177+0000 7f9573fff700 1 -- 172.21.15.91:0/4042758772 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f957c0892f0 con 0x7f958c0d2440 2023-11-24T06:08:38.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.177+0000 7f958affd700 1 --2- 172.21.15.91:0/4042758772 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f957806aed0 0x7f957806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.177+0000 7f958affd700 1 --2- 172.21.15.91:0/4042758772 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f957806aed0 0x7f957806d390 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9580014040 tx=0x7f9580008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:38.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.181+0000 7f9573fff700 1 -- 172.21.15.91:0/4042758772 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f958c0c81a0 con 0x7f958c0d2440 2023-11-24T06:08:38.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.370+0000 7f959187e700 1 -- 172.21.15.91:0/4042758772 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f958c0c81a0 con 0x7f958c0d2440 2023-11-24T06:08:38.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.371+0000 7f9573fff700 1 -- 172.21.15.91:0/4042758772 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f958c0c81a0 con 0x7f958c0d2440 2023-11-24T06:08:38.374 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 32 2023-11-24T06:08:38.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.373+0000 7f959187e700 1 -- 172.21.15.91:0/4042758772 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f957806aed0 msgr2=0x7f957806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.373+0000 7f959187e700 1 --2- 172.21.15.91:0/4042758772 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f957806aed0 0x7f957806d390 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9580014040 tx=0x7f9580008030 comp rx=0 tx=0).stop 2023-11-24T06:08:38.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.373+0000 7f959187e700 1 -- 172.21.15.91:0/4042758772 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f958c0d2440 msgr2=0x7f958c143d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.373+0000 7f959187e700 1 --2- 172.21.15.91:0/4042758772 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f958c0d2440 0x7f958c143d90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f957c00e810 tx=0x7f957c00eb20 comp rx=0 tx=0).stop 2023-11-24T06:08:38.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.373+0000 7f959187e700 1 -- 172.21.15.91:0/4042758772 shutdown_connections 2023-11-24T06:08:38.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.373+0000 7f959187e700 1 --2- 172.21.15.91:0/4042758772 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f957806aed0 0x7f957806d390 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.373+0000 7f959187e700 1 --2- 172.21.15.91:0/4042758772 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f958c15e080 0x7f958c0ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.373+0000 7f959187e700 1 --2- 172.21.15.91:0/4042758772 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f958c0d0740 0x7f958c0cc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.373+0000 7f959187e700 1 --2- 172.21.15.91:0/4042758772 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f958c0d2440 0x7f958c143d90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.373+0000 7f959187e700 1 -- 172.21.15.91:0/4042758772 >> 172.21.15.91:0/4042758772 conn(0x7f958c09d180 msgr2=0x7f958c15f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:38.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.373+0000 7f959187e700 1 -- 172.21.15.91:0/4042758772 shutdown_connections 2023-11-24T06:08:38.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.373+0000 7f959187e700 1 -- 172.21.15.91:0/4042758772 wait complete. 2023-11-24T06:08:38.388 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:08:38.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.508+0000 7fe19729c700 1 Processor -- start 2023-11-24T06:08:38.511 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.508+0000 7fe19729c700 1 -- start start 2023-11-24T06:08:38.513 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe19729c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fe19015e1b0 0x7fe190066ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.513 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe19729c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe19006afd0 0x7fe19006b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.513 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe19729c700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe19006b970 con 0x7fe1900ccb30 2023-11-24T06:08:38.513 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe19729c700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1900b5fd0 con 0x7fe19015e1b0 2023-11-24T06:08:38.513 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe19729c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1900b53c0 con 0x7fe19006afd0 2023-11-24T06:08:38.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe195038700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fe19015e1b0 0x7fe190066ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe195839700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fe1900ccb30 0x7fe1900ccf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53922/0 (socket says 172.21.15.91:53922) 2023-11-24T06:08:38.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe195038700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fe19015e1b0 0x7fe190066ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58984/0 (socket says 172.21.15.91:58984) 2023-11-24T06:08:38.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe195839700 1 -- 172.21.15.91:0/1586029819 learned_addr learned my addr 172.21.15.91:0/1586029819 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:38.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe194837700 1 --2- 172.21.15.91:0/1586029819 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe19006afd0 0x7fe19006b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe195038700 1 -- 172.21.15.91:0/1586029819 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe19006afd0 msgr2=0x7fe19006b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:38.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe195038700 1 --2- 172.21.15.91:0/1586029819 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe19006afd0 0x7fe19006b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe195038700 1 -- 172.21.15.91:0/1586029819 >> v1:172.21.15.91:6789/0 conn(0x7fe1900ccb30 legacy=0x7fe1900ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe195038700 1 -- 172.21.15.91:0/1586029819 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1900b56a0 con 0x7fe19015e1b0 2023-11-24T06:08:38.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe187fff700 1 -- 172.21.15.91:0/1586029819 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784885887 0 0) 0x7fe19006b970 con 0x7fe1900ccb30 2023-11-24T06:08:38.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe195038700 1 --2- 172.21.15.91:0/1586029819 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe19015e1b0 0x7fe190066ce0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009840 tx=0x7fe18c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba4ad2262367752 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:38.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe187fff700 1 -- 172.21.15.91:0/1586029819 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c011830 con 0x7fe19015e1b0 2023-11-24T06:08:38.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe187fff700 1 -- 172.21.15.91:0/1586029819 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18c011990 con 0x7fe19015e1b0 2023-11-24T06:08:38.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.509+0000 7fe187fff700 1 -- 172.21.15.91:0/1586029819 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c011d20 con 0x7fe19015e1b0 2023-11-24T06:08:38.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.510+0000 7fe19729c700 1 -- 172.21.15.91:0/1586029819 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe19015e1b0 msgr2=0x7fe190066ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.510+0000 7fe19729c700 1 --2- 172.21.15.91:0/1586029819 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe19015e1b0 0x7fe190066ce0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009840 tx=0x7fe18c009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:38.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.510+0000 7fe19729c700 1 -- 172.21.15.91:0/1586029819 shutdown_connections 2023-11-24T06:08:38.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.510+0000 7fe19729c700 1 --2- 172.21.15.91:0/1586029819 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe19015e1b0 0x7fe190066ce0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.510+0000 7fe19729c700 1 --2- 172.21.15.91:0/1586029819 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe19006afd0 0x7fe19006b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.510+0000 7fe19729c700 1 -- 172.21.15.91:0/1586029819 >> 172.21.15.91:0/1586029819 conn(0x7fe1900b1e80 msgr2=0x7fe1900b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:38.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.510+0000 7fe19729c700 1 -- 172.21.15.91:0/1586029819 shutdown_connections 2023-11-24T06:08:38.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.510+0000 7fe19729c700 1 -- 172.21.15.91:0/1586029819 wait complete. 2023-11-24T06:08:38.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.511+0000 7fe19729c700 1 Processor -- start 2023-11-24T06:08:38.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.511+0000 7fe19729c700 1 -- start start 2023-11-24T06:08:38.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe19729c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe19006afd0 0x7fe19009dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe19729c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1900ccb30 0x7fe190160f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe19729c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe19015e1b0 0x7fe19015faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe19729c700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19009e370 con 0x7fe19015e1b0 2023-11-24T06:08:38.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe19729c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1900c76b0 con 0x7fe1900ccb30 2023-11-24T06:08:38.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe19729c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe190161490 con 0x7fe19006afd0 2023-11-24T06:08:38.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe195839700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe19015e1b0 0x7fe19015faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe195839700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe19015e1b0 0x7fe19015faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:58994/0 (socket says 172.21.15.91:58994) 2023-11-24T06:08:38.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe195839700 1 -- 172.21.15.91:0/2268641036 learned_addr learned my addr 172.21.15.91:0/2268641036 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:38.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe195038700 1 --2- 172.21.15.91:0/2268641036 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe19006afd0 0x7fe19009dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe194837700 1 --2- 172.21.15.91:0/2268641036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1900ccb30 0x7fe190160f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe195839700 1 -- 172.21.15.91:0/2268641036 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe19006afd0 msgr2=0x7fe19009dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe195839700 1 --2- 172.21.15.91:0/2268641036 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe19006afd0 0x7fe19009dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe195839700 1 -- 172.21.15.91:0/2268641036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1900ccb30 msgr2=0x7fe190160f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe195839700 1 --2- 172.21.15.91:0/2268641036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1900ccb30 0x7fe190160f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe195839700 1 -- 172.21.15.91:0/2268641036 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18c0094a0 con 0x7fe19015e1b0 2023-11-24T06:08:38.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe195839700 1 --2- 172.21.15.91:0/2268641036 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe19015e1b0 0x7fe19015faa0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe18800cc70 tx=0x7fe18800cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:38.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.512+0000 7fe185ffb700 1 -- 172.21.15.91:0/2268641036 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18800aac0 con 0x7fe19015e1b0 2023-11-24T06:08:38.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.513+0000 7fe185ffb700 1 -- 172.21.15.91:0/2268641036 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18800ac40 con 0x7fe19015e1b0 2023-11-24T06:08:38.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.513+0000 7fe185ffb700 1 -- 172.21.15.91:0/2268641036 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1880070d0 con 0x7fe19015e1b0 2023-11-24T06:08:38.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.513+0000 7fe19729c700 1 -- 172.21.15.91:0/2268641036 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19014e040 con 0x7fe19015e1b0 2023-11-24T06:08:38.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.513+0000 7fe19729c700 1 -- 172.21.15.91:0/2268641036 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19014e600 con 0x7fe19015e1b0 2023-11-24T06:08:38.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.514+0000 7fe19729c700 1 -- 172.21.15.91:0/2268641036 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe190075480 con 0x7fe19015e1b0 2023-11-24T06:08:38.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.514+0000 7fe185ffb700 1 -- 172.21.15.91:0/2268641036 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe188007250 con 0x7fe19015e1b0 2023-11-24T06:08:38.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.514+0000 7fe185ffb700 1 --2- 172.21.15.91:0/2268641036 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe17406aed0 0x7fe17406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.514+0000 7fe185ffb700 1 -- 172.21.15.91:0/2268641036 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe188088fd0 con 0x7fe19015e1b0 2023-11-24T06:08:38.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.514+0000 7fe195038700 1 --2- 172.21.15.91:0/2268641036 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe17406aed0 0x7fe17406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.515+0000 7fe195038700 1 --2- 172.21.15.91:0/2268641036 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe17406aed0 0x7fe17406d390 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0090b0 tx=0x7fe18c01c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:38.521 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.519+0000 7fe185ffb700 1 -- 172.21.15.91:0/2268641036 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe188054080 con 0x7fe19015e1b0 2023-11-24T06:08:38.713 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.710+0000 7fe19729c700 1 -- 172.21.15.91:0/2268641036 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe1900cf990 con 0x7fe19015e1b0 2023-11-24T06:08:38.713 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.710+0000 7fe185ffb700 1 -- 172.21.15.91:0/2268641036 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe1880576a0 con 0x7fe19015e1b0 2023-11-24T06:08:38.713 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 64 2023-11-24T06:08:38.715 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.712+0000 7fe19729c700 1 -- 172.21.15.91:0/2268641036 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe17406aed0 msgr2=0x7fe17406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.716 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.712+0000 7fe19729c700 1 --2- 172.21.15.91:0/2268641036 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe17406aed0 0x7fe17406d390 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0090b0 tx=0x7fe18c01c040 comp rx=0 tx=0).stop 2023-11-24T06:08:38.716 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.712+0000 7fe19729c700 1 -- 172.21.15.91:0/2268641036 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe19015e1b0 msgr2=0x7fe19015faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.716 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.712+0000 7fe19729c700 1 --2- 172.21.15.91:0/2268641036 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe19015e1b0 0x7fe19015faa0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe18800cc70 tx=0x7fe18800cf80 comp rx=0 tx=0).stop 2023-11-24T06:08:38.716 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.712+0000 7fe19729c700 1 -- 172.21.15.91:0/2268641036 shutdown_connections 2023-11-24T06:08:38.716 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.712+0000 7fe19729c700 1 --2- 172.21.15.91:0/2268641036 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe17406aed0 0x7fe17406d390 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.716 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.712+0000 7fe19729c700 1 --2- 172.21.15.91:0/2268641036 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe19015e1b0 0x7fe19015faa0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.716 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.712+0000 7fe19729c700 1 --2- 172.21.15.91:0/2268641036 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe19006afd0 0x7fe19009dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.716 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.712+0000 7fe19729c700 1 --2- 172.21.15.91:0/2268641036 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe1900ccb30 0x7fe190160f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.716 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.712+0000 7fe19729c700 1 -- 172.21.15.91:0/2268641036 >> 172.21.15.91:0/2268641036 conn(0x7fe1900b1e80 msgr2=0x7fe1900ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:38.717 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.712+0000 7fe19729c700 1 -- 172.21.15.91:0/2268641036 shutdown_connections 2023-11-24T06:08:38.717 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.712+0000 7fe19729c700 1 -- 172.21.15.91:0/2268641036 wait complete. 2023-11-24T06:08:38.727 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T06:08:38.850 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.847+0000 7fdb72449700 1 Processor -- start 2023-11-24T06:08:38.850 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.847+0000 7fdb72449700 1 -- start start 2023-11-24T06:08:38.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.847+0000 7fdb72449700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fdb6c071bb0 0x7fdb6c0b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.847+0000 7fdb72449700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fdb6c071f90 0x7fdb6c15b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.847+0000 7fdb72449700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb6c0b4de0 con 0x7fdb6c15ba30 2023-11-24T06:08:38.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.847+0000 7fdb72449700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb6c0b3ae0 con 0x7fdb6c071f90 2023-11-24T06:08:38.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.847+0000 7fdb72449700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb6c0b3dc0 con 0x7fdb6c071bb0 2023-11-24T06:08:38.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.848+0000 7fdb6b7fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fdb6c071f90 0x7fdb6c15b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.853 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.848+0000 7fdb6b7fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fdb6c071f90 0x7fdb6c15b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59000/0 (socket says 172.21.15.91:59000) 2023-11-24T06:08:38.853 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.848+0000 7fdb6b7fe700 1 -- 172.21.15.91:0/3976815208 learned_addr learned my addr 172.21.15.91:0/3976815208 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:38.853 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.848+0000 7fdb6b7fe700 1 -- 172.21.15.91:0/3976815208 >> v2:172.21.15.94:3300/0 conn(0x7fdb6c071bb0 msgr2=0x7fdb6c0b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:38.853 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.848+0000 7fdb6b7fe700 1 --2- 172.21.15.91:0/3976815208 >> v2:172.21.15.94:3300/0 conn(0x7fdb6c071bb0 0x7fdb6c0b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.853 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.848+0000 7fdb6b7fe700 1 -- 172.21.15.91:0/3976815208 >> v1:172.21.15.94:6789/0 conn(0x7fdb6c15ba30 legacy=0x7fdb6c15c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:38.853 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.848+0000 7fdb6b7fe700 1 -- 172.21.15.91:0/3976815208 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c0b40a0 con 0x7fdb6c071f90 2023-11-24T06:08:38.853 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.848+0000 7fdb6b7fe700 1 --2- 172.21.15.91:0/3976815208 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb6c071f90 0x7fdb6c15b4f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0b3800 tx=0x7fdb6000f050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba621e3395d3e1c2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:38.854 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.848+0000 7fdb6affd700 1 -- 172.21.15.91:0/3976815208 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb60012070 con 0x7fdb6c071f90 2023-11-24T06:08:38.854 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.848+0000 7fdb6affd700 1 -- 172.21.15.91:0/3976815208 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6000f9d0 con 0x7fdb6c071f90 2023-11-24T06:08:38.854 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.848+0000 7fdb6affd700 1 -- 172.21.15.91:0/3976815208 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6000fd60 con 0x7fdb6c071f90 2023-11-24T06:08:38.854 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.849+0000 7fdb72449700 1 -- 172.21.15.91:0/3976815208 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb6c071f90 msgr2=0x7fdb6c15b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.854 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.849+0000 7fdb72449700 1 --2- 172.21.15.91:0/3976815208 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb6c071f90 0x7fdb6c15b4f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0b3800 tx=0x7fdb6000f050 comp rx=0 tx=0).stop 2023-11-24T06:08:38.854 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.849+0000 7fdb72449700 1 -- 172.21.15.91:0/3976815208 shutdown_connections 2023-11-24T06:08:38.855 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.849+0000 7fdb72449700 1 --2- 172.21.15.91:0/3976815208 >> v2:172.21.15.94:3300/0 conn(0x7fdb6c071bb0 0x7fdb6c0b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.855 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.849+0000 7fdb72449700 1 --2- 172.21.15.91:0/3976815208 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb6c071f90 0x7fdb6c15b4f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.855 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.849+0000 7fdb72449700 1 -- 172.21.15.91:0/3976815208 >> 172.21.15.91:0/3976815208 conn(0x7fdb6c0b2740 msgr2=0x7fdb6c0b2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:38.855 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.849+0000 7fdb72449700 1 -- 172.21.15.91:0/3976815208 shutdown_connections 2023-11-24T06:08:38.855 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.849+0000 7fdb72449700 1 -- 172.21.15.91:0/3976815208 wait complete. 2023-11-24T06:08:38.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.850+0000 7fdb72449700 1 Processor -- start 2023-11-24T06:08:38.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.850+0000 7fdb72449700 1 -- start start 2023-11-24T06:08:38.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.851+0000 7fdb72449700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdb6c071bb0 0x7fdb6c143690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.851+0000 7fdb72449700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb6c071f90 0x7fdb6c1efa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.851+0000 7fdb72449700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb6c15ba30 0x7fdb6c144a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.851+0000 7fdb72449700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c143bd0 con 0x7fdb6c15ba30 2023-11-24T06:08:38.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.851+0000 7fdb72449700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c145f50 con 0x7fdb6c071f90 2023-11-24T06:08:38.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.851+0000 7fdb72449700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c1eff80 con 0x7fdb6c071bb0 2023-11-24T06:08:38.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.851+0000 7fdb709e6700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb6c15ba30 0x7fdb6c144a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.851+0000 7fdb709e6700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb6c15ba30 0x7fdb6c144a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59008/0 (socket says 172.21.15.91:59008) 2023-11-24T06:08:38.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.851+0000 7fdb709e6700 1 -- 172.21.15.91:0/1038703944 learned_addr learned my addr 172.21.15.91:0/1038703944 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:38.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.851+0000 7fdb6bfff700 1 --2- 172.21.15.91:0/1038703944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdb6c071bb0 0x7fdb6c143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.851+0000 7fdb6b7fe700 1 --2- 172.21.15.91:0/1038703944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb6c071f90 0x7fdb6c1efa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.851+0000 7fdb709e6700 1 -- 172.21.15.91:0/1038703944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdb6c071bb0 msgr2=0x7fdb6c143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.851+0000 7fdb709e6700 1 --2- 172.21.15.91:0/1038703944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdb6c071bb0 0x7fdb6c143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.851+0000 7fdb709e6700 1 -- 172.21.15.91:0/1038703944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb6c071f90 msgr2=0x7fdb6c1efa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:38.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.851+0000 7fdb709e6700 1 --2- 172.21.15.91:0/1038703944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb6c071f90 0x7fdb6c1efa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:38.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.851+0000 7fdb709e6700 1 -- 172.21.15.91:0/1038703944 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6000da00 con 0x7fdb6c15ba30 2023-11-24T06:08:38.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.852+0000 7fdb709e6700 1 --2- 172.21.15.91:0/1038703944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb6c15ba30 0x7fdb6c144a50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdb6400b990 tx=0x7fdb6400bca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:38.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.852+0000 7fdb68ff9700 1 -- 172.21.15.91:0/1038703944 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6400d890 con 0x7fdb6c15ba30 2023-11-24T06:08:38.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.852+0000 7fdb68ff9700 1 -- 172.21.15.91:0/1038703944 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb6400d9f0 con 0x7fdb6c15ba30 2023-11-24T06:08:38.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.852+0000 7fdb72449700 1 -- 172.21.15.91:0/1038703944 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6c145340 con 0x7fdb6c15ba30 2023-11-24T06:08:38.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.852+0000 7fdb68ff9700 1 -- 172.21.15.91:0/1038703944 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6400dda0 con 0x7fdb6c15ba30 2023-11-24T06:08:38.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.852+0000 7fdb72449700 1 -- 172.21.15.91:0/1038703944 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6c145900 con 0x7fdb6c15ba30 2023-11-24T06:08:38.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.853+0000 7fdb72449700 1 -- 172.21.15.91:0/1038703944 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb6c1460c0 con 0x7fdb6c15ba30 2023-11-24T06:08:38.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.853+0000 7fdb68ff9700 1 -- 172.21.15.91:0/1038703944 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fdb64006d50 con 0x7fdb6c15ba30 2023-11-24T06:08:38.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.853+0000 7fdb68ff9700 1 --2- 172.21.15.91:0/1038703944 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdb5806aed0 0x7fdb5806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:38.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.854+0000 7fdb68ff9700 1 -- 172.21.15.91:0/1038703944 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb64089520 con 0x7fdb6c15ba30 2023-11-24T06:08:38.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.854+0000 7fdb6bfff700 1 --2- 172.21.15.91:0/1038703944 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdb5806aed0 0x7fdb5806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:38.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.854+0000 7fdb6bfff700 1 --2- 172.21.15.91:0/1038703944 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdb5806aed0 0x7fdb5806d390 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c009a10 tx=0x7fdb5c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:38.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:38.858+0000 7fdb68ff9700 1 -- 172.21.15.91:0/1038703944 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdb640545d0 con 0x7fdb6c15ba30 2023-11-24T06:08:39.065 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.063+0000 7fdb72449700 1 -- 172.21.15.91:0/1038703944 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdb6c068670 con 0x7fdb6c15ba30 2023-11-24T06:08:39.066 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.063+0000 7fdb68ff9700 1 -- 172.21.15.91:0/1038703944 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fdb64057bf0 con 0x7fdb6c15ba30 2023-11-24T06:08:39.068 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.065+0000 7fdb72449700 1 -- 172.21.15.91:0/1038703944 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdb5806aed0 msgr2=0x7fdb5806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:39.068 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.065+0000 7fdb72449700 1 --2- 172.21.15.91:0/1038703944 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdb5806aed0 0x7fdb5806d390 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c009a10 tx=0x7fdb5c009190 comp rx=0 tx=0).stop 2023-11-24T06:08:39.068 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.065+0000 7fdb72449700 1 -- 172.21.15.91:0/1038703944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb6c15ba30 msgr2=0x7fdb6c144a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:39.068 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.065+0000 7fdb72449700 1 --2- 172.21.15.91:0/1038703944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb6c15ba30 0x7fdb6c144a50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdb6400b990 tx=0x7fdb6400bca0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.068 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.065+0000 7fdb72449700 1 -- 172.21.15.91:0/1038703944 shutdown_connections 2023-11-24T06:08:39.069 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.065+0000 7fdb72449700 1 --2- 172.21.15.91:0/1038703944 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fdb5806aed0 0x7fdb5806d390 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.069 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.065+0000 7fdb72449700 1 --2- 172.21.15.91:0/1038703944 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fdb6c15ba30 0x7fdb6c144a50 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.069 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.065+0000 7fdb72449700 1 --2- 172.21.15.91:0/1038703944 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fdb6c071bb0 0x7fdb6c143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.069 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.065+0000 7fdb72449700 1 --2- 172.21.15.91:0/1038703944 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fdb6c071f90 0x7fdb6c1efa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.069 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.065+0000 7fdb72449700 1 -- 172.21.15.91:0/1038703944 >> 172.21.15.91:0/1038703944 conn(0x7fdb6c0b2740 msgr2=0x7fdb6c159710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:39.069 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.065+0000 7fdb72449700 1 -- 172.21.15.91:0/1038703944 shutdown_connections 2023-11-24T06:08:39.069 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.065+0000 7fdb72449700 1 -- 172.21.15.91:0/1038703944 wait complete. 2023-11-24T06:08:39.069 INFO:teuthology.orchestra.run.smithi091.stderr:listed 0 entries 2023-11-24T06:08:39.080 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T06:08:39.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:39.080 DEBUG:teuthology.orchestra.run.smithi094:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2023-11-24T06:08:39.127 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi094.front.sepia.ceph.com 2023-11-24T06:08:39.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:39.127 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T06:08:39.156 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:08:39.156 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:08:39.173 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:08:39.173 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:08:39.230 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:08:39.231 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:08:39.288 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:39.448 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.445+0000 7f30d5da4700 1 Processor -- start 2023-11-24T06:08:39.448 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.445+0000 7f30d5da4700 1 -- start start 2023-11-24T06:08:39.450 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.446+0000 7f30d5da4700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f30d015a440 0x7f30d00cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:39.450 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.446+0000 7f30d5da4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f30d015eb80 0x7f30d00d04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:39.450 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.446+0000 7f30d5da4700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d00b05f0 con 0x7f30d015a820 2023-11-24T06:08:39.451 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.446+0000 7f30d5da4700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30d015f080 con 0x7f30d015a440 2023-11-24T06:08:39.451 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.446+0000 7f30d5da4700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d00afd50 con 0x7f30d015eb80 2023-11-24T06:08:39.451 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.446+0000 7f30cf7fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f30d015a440 0x7f30d00cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:39.451 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.446+0000 7f30ceffd700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f30d015a820 0x7f30d00726a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53948/0 (socket says 172.21.15.91:53948) 2023-11-24T06:08:39.451 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.446+0000 7f30ceffd700 1 -- 172.21.15.91:0/837793999 learned_addr learned my addr 172.21.15.91:0/837793999 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:39.451 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.446+0000 7f30cf7fe700 1 -- 172.21.15.91:0/837793999 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f30d015eb80 msgr2=0x7f30d00d04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:39.451 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.446+0000 7f30cf7fe700 1 --2- 172.21.15.91:0/837793999 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f30d015eb80 0x7f30d00d04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.451 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.446+0000 7f30cf7fe700 1 -- 172.21.15.91:0/837793999 >> v1:172.21.15.91:6789/0 conn(0x7f30d015a820 legacy=0x7f30d00726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:39.451 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.446+0000 7f30ce7fc700 1 -- 172.21.15.91:0/837793999 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026249790 0 0) 0x7f30d00b05f0 con 0x7f30d015a820 2023-11-24T06:08:39.451 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.446+0000 7f30cf7fe700 1 -- 172.21.15.91:0/837793999 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d00b0030 con 0x7f30d015a440 2023-11-24T06:08:39.451 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.446+0000 7f30cf7fe700 1 --2- 172.21.15.91:0/837793999 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f30d015a440 0x7f30d00cffa0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f30c400a9e0 tx=0x7f30c400acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbc9c217c6a95ed server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:39.451 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.446+0000 7f30ce7fc700 1 -- 172.21.15.91:0/837793999 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c400cb70 con 0x7f30d015a440 2023-11-24T06:08:39.451 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.447+0000 7f30ce7fc700 1 -- 172.21.15.91:0/837793999 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c400ccd0 con 0x7f30d015a440 2023-11-24T06:08:39.452 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.447+0000 7f30ce7fc700 1 -- 172.21.15.91:0/837793999 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c4018620 con 0x7f30d015a440 2023-11-24T06:08:39.452 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.447+0000 7f30d5da4700 1 -- 172.21.15.91:0/837793999 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f30d015a440 msgr2=0x7f30d00cffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:39.452 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.447+0000 7f30d5da4700 1 --2- 172.21.15.91:0/837793999 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f30d015a440 0x7f30d00cffa0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f30c400a9e0 tx=0x7f30c400acf0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.452 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.447+0000 7f30d5da4700 1 -- 172.21.15.91:0/837793999 shutdown_connections 2023-11-24T06:08:39.452 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.447+0000 7f30d5da4700 1 --2- 172.21.15.91:0/837793999 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f30d015eb80 0x7f30d00d04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.452 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.447+0000 7f30d5da4700 1 --2- 172.21.15.91:0/837793999 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f30d015a440 0x7f30d00cffa0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.452 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.447+0000 7f30d5da4700 1 -- 172.21.15.91:0/837793999 >> 172.21.15.91:0/837793999 conn(0x7f30d00b5070 msgr2=0x7f30d00b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:39.452 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.447+0000 7f30d5da4700 1 -- 172.21.15.91:0/837793999 shutdown_connections 2023-11-24T06:08:39.452 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.447+0000 7f30d5da4700 1 -- 172.21.15.91:0/837793999 wait complete. 2023-11-24T06:08:39.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.448+0000 7f30d5da4700 1 Processor -- start 2023-11-24T06:08:39.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.448+0000 7f30d5da4700 1 -- start start 2023-11-24T06:08:39.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.449+0000 7f30d5da4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f30d015a440 0x7f30d0159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:39.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.449+0000 7f30d5da4700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f30d015eb80 0x7f30d015d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:39.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.449+0000 7f30d5da4700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30d0157d60 0x7f30d00ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:39.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.449+0000 7f30d5da4700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d0158ee0 con 0x7f30d015eb80 2023-11-24T06:08:39.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.449+0000 7f30d5da4700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d01f4390 con 0x7f30d0157d60 2023-11-24T06:08:39.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.449+0000 7f30d5da4700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d0158260 con 0x7f30d015a440 2023-11-24T06:08:39.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.449+0000 7f30ceffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f30d015eb80 0x7f30d015d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:39.456 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.449+0000 7f30ceffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f30d015eb80 0x7f30d015d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59020/0 (socket says 172.21.15.91:59020) 2023-11-24T06:08:39.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.449+0000 7f30ceffd700 1 -- 172.21.15.91:0/1511586846 learned_addr learned my addr 172.21.15.91:0/1511586846 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:39.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.449+0000 7f30cf7fe700 1 --2- 172.21.15.91:0/1511586846 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f30d015a440 0x7f30d0159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:39.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.449+0000 7f30cffff700 1 --2- 172.21.15.91:0/1511586846 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30d0157d60 0x7f30d00ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:39.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.450+0000 7f30ceffd700 1 -- 172.21.15.91:0/1511586846 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f30d015a440 msgr2=0x7f30d0159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:39.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.450+0000 7f30ceffd700 1 --2- 172.21.15.91:0/1511586846 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f30d015a440 0x7f30d0159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.450+0000 7f30ceffd700 1 -- 172.21.15.91:0/1511586846 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30d0157d60 msgr2=0x7f30d00ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:39.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.450+0000 7f30ceffd700 1 --2- 172.21.15.91:0/1511586846 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30d0157d60 0x7f30d00ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.450+0000 7f30ceffd700 1 -- 172.21.15.91:0/1511586846 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c400a640 con 0x7f30d015eb80 2023-11-24T06:08:39.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.450+0000 7f30cffff700 1 --2- 172.21.15.91:0/1511586846 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30d0157d60 0x7f30d00ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:39.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.450+0000 7f30ceffd700 1 --2- 172.21.15.91:0/1511586846 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f30d015eb80 0x7f30d015d1d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f30c000c540 tx=0x7f30c000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:39.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.450+0000 7f30b7fff700 1 -- 172.21.15.91:0/1511586846 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c0004c80 con 0x7f30d015eb80 2023-11-24T06:08:39.457 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.450+0000 7f30b7fff700 1 -- 172.21.15.91:0/1511586846 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c0004de0 con 0x7f30d015eb80 2023-11-24T06:08:39.458 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.450+0000 7f30d5da4700 1 -- 172.21.15.91:0/1511586846 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d0158540 con 0x7f30d015eb80 2023-11-24T06:08:39.458 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.450+0000 7f30b7fff700 1 -- 172.21.15.91:0/1511586846 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c0007310 con 0x7f30d015eb80 2023-11-24T06:08:39.458 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.450+0000 7f30d5da4700 1 -- 172.21.15.91:0/1511586846 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d0158b00 con 0x7f30d015eb80 2023-11-24T06:08:39.458 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.451+0000 7f30d5da4700 1 -- 172.21.15.91:0/1511586846 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d0159150 con 0x7f30d015eb80 2023-11-24T06:08:39.458 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.451+0000 7f30b7fff700 1 -- 172.21.15.91:0/1511586846 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f30c0007470 con 0x7f30d015eb80 2023-11-24T06:08:39.458 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.452+0000 7f30b7fff700 1 --2- 172.21.15.91:0/1511586846 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f30bc06af00 0x7f30bc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:39.458 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.452+0000 7f30b7fff700 1 -- 172.21.15.91:0/1511586846 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30c00892f0 con 0x7f30d015eb80 2023-11-24T06:08:39.458 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.452+0000 7f30cf7fe700 1 --2- 172.21.15.91:0/1511586846 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f30bc06af00 0x7f30bc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:39.458 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.452+0000 7f30cf7fe700 1 --2- 172.21.15.91:0/1511586846 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f30bc06af00 0x7f30bc06d3c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f30c400a1c0 tx=0x7f30c4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:39.459 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.456+0000 7f30b7fff700 1 -- 172.21.15.91:0/1511586846 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f30c00543b0 con 0x7f30d015eb80 2023-11-24T06:08:39.652 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.649+0000 7f30d5da4700 1 -- 172.21.15.91:0/1511586846 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30d0066ce0 con 0x7f30d015eb80 2023-11-24T06:08:39.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.650+0000 7f30b7fff700 1 -- 172.21.15.91:0/1511586846 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7f30c00579d0 con 0x7f30d015eb80 2023-11-24T06:08:39.653 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:39.653 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":20,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:02.415612+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:39.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.652+0000 7f30d5da4700 1 -- 172.21.15.91:0/1511586846 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f30bc06af00 msgr2=0x7f30bc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:39.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.652+0000 7f30d5da4700 1 --2- 172.21.15.91:0/1511586846 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f30bc06af00 0x7f30bc06d3c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f30c400a1c0 tx=0x7f30c4008040 comp rx=0 tx=0).stop 2023-11-24T06:08:39.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.652+0000 7f30d5da4700 1 -- 172.21.15.91:0/1511586846 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f30d015eb80 msgr2=0x7f30d015d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:39.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.652+0000 7f30d5da4700 1 --2- 172.21.15.91:0/1511586846 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f30d015eb80 0x7f30d015d1d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f30c000c540 tx=0x7f30c000c850 comp rx=0 tx=0).stop 2023-11-24T06:08:39.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.652+0000 7f30d5da4700 1 -- 172.21.15.91:0/1511586846 shutdown_connections 2023-11-24T06:08:39.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.652+0000 7f30d5da4700 1 --2- 172.21.15.91:0/1511586846 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f30bc06af00 0x7f30bc06d3c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.652+0000 7f30d5da4700 1 --2- 172.21.15.91:0/1511586846 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f30d0157d60 0x7f30d00ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.652+0000 7f30d5da4700 1 --2- 172.21.15.91:0/1511586846 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f30d015a440 0x7f30d0159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.652+0000 7f30d5da4700 1 --2- 172.21.15.91:0/1511586846 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f30d015eb80 0x7f30d015d1d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.652+0000 7f30d5da4700 1 -- 172.21.15.91:0/1511586846 >> 172.21.15.91:0/1511586846 conn(0x7f30d00b5070 msgr2=0x7f30d006d350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:39.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.652+0000 7f30d5da4700 1 -- 172.21.15.91:0/1511586846 shutdown_connections 2023-11-24T06:08:39.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.652+0000 7f30d5da4700 1 -- 172.21.15.91:0/1511586846 wait complete. 2023-11-24T06:08:39.667 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:07:58.710106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:07:59.774819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:08:39.668 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:08:39.791 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.788+0000 7f154226f700 1 Processor -- start 2023-11-24T06:08:39.791 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.788+0000 7f154226f700 1 -- start start 2023-11-24T06:08:39.793 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.789+0000 7f154226f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f153c06b620 0x7f153c0b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:39.793 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.789+0000 7f154226f700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f153c06ee60 0x7f153c0b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:39.793 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.789+0000 7f154226f700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c071f50 con 0x7f153c0b5ab0 2023-11-24T06:08:39.793 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.789+0000 7f154226f700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153c072230 con 0x7f153c06ee60 2023-11-24T06:08:39.793 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.789+0000 7f154226f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153c072510 con 0x7f153c06b620 2023-11-24T06:08:39.793 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.789+0000 7f154080c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f153c06ee60 0x7f153c0b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:39.793 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.789+0000 7f154080c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f153c06ee60 0x7f153c0b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59030/0 (socket says 172.21.15.91:59030) 2023-11-24T06:08:39.793 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.789+0000 7f154080c700 1 -- 172.21.15.91:0/1896591903 learned_addr learned my addr 172.21.15.91:0/1896591903 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:39.793 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.789+0000 7f153bfff700 1 --2- 172.21.15.91:0/1896591903 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f153c06b620 0x7f153c0b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:39.793 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.789+0000 7f154080c700 1 -- 172.21.15.91:0/1896591903 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f153c06b620 msgr2=0x7f153c0b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:39.794 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.789+0000 7f154080c700 1 --2- 172.21.15.91:0/1896591903 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f153c06b620 0x7f153c0b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.794 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.789+0000 7f154080c700 1 -- 172.21.15.91:0/1896591903 >> v1:172.21.15.94:6789/0 conn(0x7f153c0b5ab0 legacy=0x7f153c161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:39.794 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.789+0000 7f154080c700 1 -- 172.21.15.91:0/1896591903 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c0727f0 con 0x7f153c06ee60 2023-11-24T06:08:39.794 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.789+0000 7f154080c700 1 --2- 172.21.15.91:0/1896591903 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f153c06ee60 0x7f153c0b3f80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1534009e50 tx=0x7f153400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1276e5902e0c758 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:39.795 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.793+0000 7f153affd700 1 -- 172.21.15.91:0/1896591903 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153400e070 con 0x7f153c06ee60 2023-11-24T06:08:39.796 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.793+0000 7f153affd700 1 -- 172.21.15.91:0/1896591903 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153400b910 con 0x7f153c06ee60 2023-11-24T06:08:39.796 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.793+0000 7f153affd700 1 -- 172.21.15.91:0/1896591903 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153400bcc0 con 0x7f153c06ee60 2023-11-24T06:08:39.796 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.793+0000 7f154226f700 1 -- 172.21.15.91:0/1896591903 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f153c06ee60 msgr2=0x7f153c0b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:39.796 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.793+0000 7f154226f700 1 --2- 172.21.15.91:0/1896591903 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f153c06ee60 0x7f153c0b3f80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1534009e50 tx=0x7f153400b330 comp rx=0 tx=0).stop 2023-11-24T06:08:39.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.793+0000 7f154226f700 1 -- 172.21.15.91:0/1896591903 shutdown_connections 2023-11-24T06:08:39.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.793+0000 7f154226f700 1 --2- 172.21.15.91:0/1896591903 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f153c06ee60 0x7f153c0b3f80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.793+0000 7f154226f700 1 --2- 172.21.15.91:0/1896591903 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f153c06b620 0x7f153c0b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.794+0000 7f154226f700 1 -- 172.21.15.91:0/1896591903 >> 172.21.15.91:0/1896591903 conn(0x7f153c0b2740 msgr2=0x7f153c0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:39.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.794+0000 7f154226f700 1 -- 172.21.15.91:0/1896591903 shutdown_connections 2023-11-24T06:08:39.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.794+0000 7f154226f700 1 -- 172.21.15.91:0/1896591903 wait complete. 2023-11-24T06:08:39.797 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.795+0000 7f154226f700 1 Processor -- start 2023-11-24T06:08:39.798 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.795+0000 7f154226f700 1 -- start start 2023-11-24T06:08:39.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f154226f700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f153c06b620 0x7f153c0b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:39.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f154226f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f153c06ee60 0x7f153c145f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:39.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f154226f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f153c0b5ab0 0x7f153c1464d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:39.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f154226f700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c1f7310 con 0x7f153c06b620 2023-11-24T06:08:39.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f154226f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c1f4a80 con 0x7f153c06ee60 2023-11-24T06:08:39.800 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f154226f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153c1f7520 con 0x7f153c0b5ab0 2023-11-24T06:08:39.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f153bfff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f153c06b620 0x7f153c0b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:39.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f153bfff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f153c06b620 0x7f153c0b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59032/0 (socket says 172.21.15.91:59032) 2023-11-24T06:08:39.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f153bfff700 1 -- 172.21.15.91:0/3494273005 learned_addr learned my addr 172.21.15.91:0/3494273005 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:39.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f154080c700 1 --2- 172.21.15.91:0/3494273005 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f153c0b5ab0 0x7f153c1464d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:39.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f153b7fe700 1 --2- 172.21.15.91:0/3494273005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f153c06ee60 0x7f153c145f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:39.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f153bfff700 1 -- 172.21.15.91:0/3494273005 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f153c0b5ab0 msgr2=0x7f153c1464d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:39.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f153bfff700 1 --2- 172.21.15.91:0/3494273005 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f153c0b5ab0 0x7f153c1464d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f153bfff700 1 -- 172.21.15.91:0/3494273005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f153c06ee60 msgr2=0x7f153c145f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:39.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f153bfff700 1 --2- 172.21.15.91:0/3494273005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f153c06ee60 0x7f153c145f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f153bfff700 1 -- 172.21.15.91:0/3494273005 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1534009b30 con 0x7f153c06b620 2023-11-24T06:08:39.801 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f154080c700 1 --2- 172.21.15.91:0/3494273005 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f153c0b5ab0 0x7f153c1464d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:39.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f153b7fe700 1 --2- 172.21.15.91:0/3494273005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f153c06ee60 0x7f153c145f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:39.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f153bfff700 1 --2- 172.21.15.91:0/3494273005 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f153c06b620 0x7f153c0b1af0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f152400e9b0 tx=0x7f152400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:39.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f1538ff9700 1 -- 172.21.15.91:0/3494273005 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1524009710 con 0x7f153c06b620 2023-11-24T06:08:39.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f1538ff9700 1 -- 172.21.15.91:0/3494273005 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152400c850 con 0x7f153c06b620 2023-11-24T06:08:39.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f154226f700 1 -- 172.21.15.91:0/3494273005 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153c1f7800 con 0x7f153c06b620 2023-11-24T06:08:39.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f1538ff9700 1 -- 172.21.15.91:0/3494273005 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152400cbc0 con 0x7f153c06b620 2023-11-24T06:08:39.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.796+0000 7f154226f700 1 -- 172.21.15.91:0/3494273005 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153c0b2030 con 0x7f153c06b620 2023-11-24T06:08:39.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.798+0000 7f154226f700 1 -- 172.21.15.91:0/3494273005 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153c1f7e50 con 0x7f153c06b620 2023-11-24T06:08:39.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.798+0000 7f1538ff9700 1 -- 172.21.15.91:0/3494273005 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f152400cd20 con 0x7f153c06b620 2023-11-24T06:08:39.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.798+0000 7f1538ff9700 1 --2- 172.21.15.91:0/3494273005 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f151c06aed0 0x7f151c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:39.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.798+0000 7f153b7fe700 1 --2- 172.21.15.91:0/3494273005 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f151c06aed0 0x7f151c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:39.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.798+0000 7f1538ff9700 1 -- 172.21.15.91:0/3494273005 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1524089720 con 0x7f153c06b620 2023-11-24T06:08:39.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.799+0000 7f153b7fe700 1 --2- 172.21.15.91:0/3494273005 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f151c06aed0 0x7f151c06d390 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f153c1e6320 tx=0x7f152c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:39.805 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.803+0000 7f1538ff9700 1 -- 172.21.15.91:0/3494273005 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15240547d0 con 0x7f153c06b620 2023-11-24T06:08:39.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.993+0000 7f154226f700 1 -- 172.21.15.91:0/3494273005 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f153c1f7660 con 0x7f153c06b620 2023-11-24T06:08:39.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.993+0000 7f1538ff9700 1 -- 172.21.15.91:0/3494273005 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1524021020 con 0x7f153c06b620 2023-11-24T06:08:39.996 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:08:39.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.996+0000 7f154226f700 1 -- 172.21.15.91:0/3494273005 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f151c06aed0 msgr2=0x7f151c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:39.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.996+0000 7f154226f700 1 --2- 172.21.15.91:0/3494273005 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f151c06aed0 0x7f151c06d390 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f153c1e6320 tx=0x7f152c009040 comp rx=0 tx=0).stop 2023-11-24T06:08:39.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.996+0000 7f154226f700 1 -- 172.21.15.91:0/3494273005 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f153c06b620 msgr2=0x7f153c0b1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:39.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.996+0000 7f154226f700 1 --2- 172.21.15.91:0/3494273005 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f153c06b620 0x7f153c0b1af0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f152400e9b0 tx=0x7f152400ed70 comp rx=0 tx=0).stop 2023-11-24T06:08:39.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.996+0000 7f154226f700 1 -- 172.21.15.91:0/3494273005 shutdown_connections 2023-11-24T06:08:39.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.996+0000 7f154226f700 1 --2- 172.21.15.91:0/3494273005 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f151c06aed0 0x7f151c06d390 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.996+0000 7f154226f700 1 --2- 172.21.15.91:0/3494273005 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f153c0b5ab0 0x7f153c1464d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.996+0000 7f154226f700 1 --2- 172.21.15.91:0/3494273005 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f153c06b620 0x7f153c0b1af0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.996+0000 7f154226f700 1 --2- 172.21.15.91:0/3494273005 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f153c06ee60 0x7f153c145f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:39.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.996+0000 7f154226f700 1 -- 172.21.15.91:0/3494273005 >> 172.21.15.91:0/3494273005 conn(0x7f153c0b2740 msgr2=0x7f153c147660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:40.000 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.996+0000 7f154226f700 1 -- 172.21.15.91:0/3494273005 shutdown_connections 2023-11-24T06:08:40.000 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:39.996+0000 7f154226f700 1 -- 172.21.15.91:0/3494273005 wait complete. 2023-11-24T06:08:40.011 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:08:40.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.130+0000 7f15fc3a9700 1 Processor -- start 2023-11-24T06:08:40.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.130+0000 7f15fc3a9700 1 -- start start 2023-11-24T06:08:40.135 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.131+0000 7f15fc3a9700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f15f40722c0 0x7f15f40b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.136 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.131+0000 7f15fc3a9700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f4143ec0 con 0x7f15f41440f0 2023-11-24T06:08:40.136 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.131+0000 7f15fc3a9700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f40b3ae0 con 0x7f15f4143ae0 2023-11-24T06:08:40.136 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.131+0000 7f15fc3a9700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15f40b40a0 con 0x7f15f40722c0 2023-11-24T06:08:40.136 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.131+0000 7f15fa946700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f15f41440f0 0x7f15f406e510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:53978/0 (socket says 172.21.15.91:53978) 2023-11-24T06:08:40.136 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.131+0000 7f15fa946700 1 -- 172.21.15.91:0/1503302165 learned_addr learned my addr 172.21.15.91:0/1503302165 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:40.136 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.132+0000 7f15f9944700 1 --2- 172.21.15.91:0/1503302165 >> v2:172.21.15.94:3300/0 conn(0x7f15f40722c0 0x7f15f40b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.136 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.132+0000 7f15f9143700 1 -- 172.21.15.91:0/1503302165 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920124520 0 0) 0x7f15f4143ec0 con 0x7f15f41440f0 2023-11-24T06:08:40.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.132+0000 7f15f9143700 1 -- 172.21.15.91:0/1503302165 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e00035b0 con 0x7f15f41440f0 2023-11-24T06:08:40.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.132+0000 7f15f9944700 1 -- 172.21.15.91:0/1503302165 >> v1:172.21.15.94:6789/0 conn(0x7f15f4143ae0 legacy=0x7f15f4071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.132+0000 7f15f9944700 1 -- 172.21.15.91:0/1503302165 >> v1:172.21.15.91:6789/0 conn(0x7f15f41440f0 legacy=0x7f15f406e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.132+0000 7f15f9944700 1 -- 172.21.15.91:0/1503302165 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f40b3dc0 con 0x7f15f40722c0 2023-11-24T06:08:40.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.132+0000 7f15f9944700 1 --2- 172.21.15.91:0/1503302165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15f40722c0 0x7f15f40b51b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009840 tx=0x7f15f0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=750f15588e4cc5f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:40.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.132+0000 7f15f9143700 1 -- 172.21.15.91:0/1503302165 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f0011830 con 0x7f15f40722c0 2023-11-24T06:08:40.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.132+0000 7f15f9143700 1 -- 172.21.15.91:0/1503302165 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f0011990 con 0x7f15f40722c0 2023-11-24T06:08:40.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.132+0000 7f15f9143700 1 -- 172.21.15.91:0/1503302165 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f0011d00 con 0x7f15f40722c0 2023-11-24T06:08:40.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.133+0000 7f15fc3a9700 1 -- 172.21.15.91:0/1503302165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15f40722c0 msgr2=0x7f15f40b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.133+0000 7f15fc3a9700 1 --2- 172.21.15.91:0/1503302165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15f40722c0 0x7f15f40b51b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009840 tx=0x7f15f0009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:40.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.133+0000 7f15fc3a9700 1 -- 172.21.15.91:0/1503302165 shutdown_connections 2023-11-24T06:08:40.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.133+0000 7f15fc3a9700 1 --2- 172.21.15.91:0/1503302165 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15f40722c0 0x7f15f40b51b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.133+0000 7f15fc3a9700 1 -- 172.21.15.91:0/1503302165 >> 172.21.15.91:0/1503302165 conn(0x7f15f40b2740 msgr2=0x7f15f40b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:40.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.133+0000 7f15fc3a9700 1 -- 172.21.15.91:0/1503302165 shutdown_connections 2023-11-24T06:08:40.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.133+0000 7f15fc3a9700 1 -- 172.21.15.91:0/1503302165 wait complete. 2023-11-24T06:08:40.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.134+0000 7f15fc3a9700 1 Processor -- start 2023-11-24T06:08:40.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.134+0000 7f15fc3a9700 1 -- start start 2023-11-24T06:08:40.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.135+0000 7f15fc3a9700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15f40722c0 0x7f15f4156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.135+0000 7f15fc3a9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15f4143ae0 0x7f15f4141fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.135+0000 7f15fc3a9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15f41440f0 0x7f15f40b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.135+0000 7f15fc3a9700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f4140310 con 0x7f15f40722c0 2023-11-24T06:08:40.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.135+0000 7f15fc3a9700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f41405f0 con 0x7f15f41440f0 2023-11-24T06:08:40.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.135+0000 7f15fc3a9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f41408d0 con 0x7f15f4143ae0 2023-11-24T06:08:40.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.135+0000 7f15fa145700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15f40722c0 0x7f15f4156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.135+0000 7f15fa145700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15f40722c0 0x7f15f4156bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59046/0 (socket says 172.21.15.91:59046) 2023-11-24T06:08:40.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.135+0000 7f15fa145700 1 -- 172.21.15.91:0/3070166028 learned_addr learned my addr 172.21.15.91:0/3070166028 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:40.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.136+0000 7f15fa946700 1 --2- 172.21.15.91:0/3070166028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15f41440f0 0x7f15f40b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.136+0000 7f15f9944700 1 --2- 172.21.15.91:0/3070166028 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15f4143ae0 0x7f15f4141fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.136+0000 7f15fa145700 1 -- 172.21.15.91:0/3070166028 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15f4143ae0 msgr2=0x7f15f4141fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.136+0000 7f15fa145700 1 --2- 172.21.15.91:0/3070166028 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15f4143ae0 0x7f15f4141fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.136+0000 7f15fa145700 1 -- 172.21.15.91:0/3070166028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15f41440f0 msgr2=0x7f15f40b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.136+0000 7f15fa145700 1 --2- 172.21.15.91:0/3070166028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15f41440f0 0x7f15f40b41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.136+0000 7f15fa145700 1 -- 172.21.15.91:0/3070166028 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f00094a0 con 0x7f15f40722c0 2023-11-24T06:08:40.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.136+0000 7f15fa946700 1 --2- 172.21.15.91:0/3070166028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15f41440f0 0x7f15f40b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:40.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.136+0000 7f15fa145700 1 --2- 172.21.15.91:0/3070166028 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15f40722c0 0x7f15f4156bf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f15e400c490 tx=0x7f15e400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:40.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.136+0000 7f15eaffd700 1 -- 172.21.15.91:0/3070166028 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e40052d0 con 0x7f15f40722c0 2023-11-24T06:08:40.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.136+0000 7f15eaffd700 1 -- 172.21.15.91:0/3070166028 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e4005450 con 0x7f15f40722c0 2023-11-24T06:08:40.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.136+0000 7f15fc3a9700 1 -- 172.21.15.91:0/3070166028 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f4140bb0 con 0x7f15f40722c0 2023-11-24T06:08:40.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.136+0000 7f15eaffd700 1 -- 172.21.15.91:0/3070166028 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e4006aa0 con 0x7f15f40722c0 2023-11-24T06:08:40.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.136+0000 7f15fc3a9700 1 -- 172.21.15.91:0/3070166028 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f41f1d60 con 0x7f15f40722c0 2023-11-24T06:08:40.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.137+0000 7f15fc3a9700 1 -- 172.21.15.91:0/3070166028 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f41f1f50 con 0x7f15f40722c0 2023-11-24T06:08:40.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.138+0000 7f15eaffd700 1 -- 172.21.15.91:0/3070166028 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f15e4003680 con 0x7f15f40722c0 2023-11-24T06:08:40.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.138+0000 7f15eaffd700 1 --2- 172.21.15.91:0/3070166028 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15d806aed0 0x7f15d806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.138+0000 7f15eaffd700 1 -- 172.21.15.91:0/3070166028 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15e4088880 con 0x7f15f40722c0 2023-11-24T06:08:40.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.138+0000 7f15f9944700 1 --2- 172.21.15.91:0/3070166028 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15d806aed0 0x7f15d806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.138+0000 7f15f9944700 1 --2- 172.21.15.91:0/3070166028 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15d806aed0 0x7f15d806d390 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f15f0000c00 tx=0x7f15f0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:40.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.143+0000 7f15eaffd700 1 -- 172.21.15.91:0/3070166028 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15f41f1f50 con 0x7f15f40722c0 2023-11-24T06:08:40.334 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.331+0000 7f15fc3a9700 1 -- 172.21.15.91:0/3070166028 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f15f41f1f50 con 0x7f15f40722c0 2023-11-24T06:08:40.334 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.331+0000 7f15eaffd700 1 -- 172.21.15.91:0/3070166028 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f15e4053940 con 0x7f15f40722c0 2023-11-24T06:08:40.334 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 32 2023-11-24T06:08:40.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.333+0000 7f15fc3a9700 1 -- 172.21.15.91:0/3070166028 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15d806aed0 msgr2=0x7f15d806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.333+0000 7f15fc3a9700 1 --2- 172.21.15.91:0/3070166028 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15d806aed0 0x7f15d806d390 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f15f0000c00 tx=0x7f15f0008030 comp rx=0 tx=0).stop 2023-11-24T06:08:40.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.333+0000 7f15fc3a9700 1 -- 172.21.15.91:0/3070166028 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15f40722c0 msgr2=0x7f15f4156bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.333+0000 7f15fc3a9700 1 --2- 172.21.15.91:0/3070166028 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15f40722c0 0x7f15f4156bf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f15e400c490 tx=0x7f15e400c850 comp rx=0 tx=0).stop 2023-11-24T06:08:40.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.333+0000 7f15fc3a9700 1 -- 172.21.15.91:0/3070166028 shutdown_connections 2023-11-24T06:08:40.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.333+0000 7f15fc3a9700 1 --2- 172.21.15.91:0/3070166028 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15d806aed0 0x7f15d806d390 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.333+0000 7f15fc3a9700 1 --2- 172.21.15.91:0/3070166028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f15f41440f0 0x7f15f40b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.333+0000 7f15fc3a9700 1 --2- 172.21.15.91:0/3070166028 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f15f40722c0 0x7f15f4156bf0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.333+0000 7f15fc3a9700 1 --2- 172.21.15.91:0/3070166028 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f15f4143ae0 0x7f15f4141fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.333+0000 7f15fc3a9700 1 -- 172.21.15.91:0/3070166028 >> 172.21.15.91:0/3070166028 conn(0x7f15f40b2740 msgr2=0x7f15f40cf910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:40.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.333+0000 7f15fc3a9700 1 -- 172.21.15.91:0/3070166028 shutdown_connections 2023-11-24T06:08:40.338 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.333+0000 7f15fc3a9700 1 -- 172.21.15.91:0/3070166028 wait complete. 2023-11-24T06:08:40.348 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:08:40.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.469+0000 7f93faa8a700 1 Processor -- start 2023-11-24T06:08:40.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.469+0000 7f93faa8a700 1 -- start start 2023-11-24T06:08:40.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.470+0000 7f93faa8a700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f93f415ad70 0x7f93f40b5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.470+0000 7f93faa8a700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f93f40adda0 0x7f93f406b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.470+0000 7f93faa8a700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f415b150 con 0x7f93f40b03b0 2023-11-24T06:08:40.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.470+0000 7f93faa8a700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93f4073610 con 0x7f93f415ad70 2023-11-24T06:08:40.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.470+0000 7f93faa8a700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93f40b6380 con 0x7f93f40adda0 2023-11-24T06:08:40.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.470+0000 7f93f8826700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f93f415ad70 0x7f93f40b5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.470+0000 7f93f8826700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f93f415ad70 0x7f93f40b5e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59054/0 (socket says 172.21.15.91:59054) 2023-11-24T06:08:40.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.470+0000 7f93f8826700 1 -- 172.21.15.91:0/3357773339 learned_addr learned my addr 172.21.15.91:0/3357773339 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:40.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.470+0000 7f93f9027700 1 --2- 172.21.15.91:0/3357773339 >> v2:172.21.15.94:3300/0 conn(0x7f93f40adda0 0x7f93f406b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.471+0000 7f93f8826700 1 -- 172.21.15.91:0/3357773339 >> v2:172.21.15.94:3300/0 conn(0x7f93f40adda0 msgr2=0x7f93f406b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:40.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.471+0000 7f93f8826700 1 --2- 172.21.15.91:0/3357773339 >> v2:172.21.15.94:3300/0 conn(0x7f93f40adda0 0x7f93f406b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.471+0000 7f93f8826700 1 -- 172.21.15.91:0/3357773339 >> v1:172.21.15.94:6789/0 conn(0x7f93f40b03b0 legacy=0x7f93f40ad690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:40.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.471+0000 7f93f8826700 1 -- 172.21.15.91:0/3357773339 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f40b0ec0 con 0x7f93f415ad70 2023-11-24T06:08:40.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.471+0000 7f93f8826700 1 --2- 172.21.15.91:0/3357773339 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93f415ad70 0x7f93f40b5e40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f93e800bbc0 tx=0x7f93e800bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595c4c808fd540e7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:40.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.471+0000 7f93f37fe700 1 -- 172.21.15.91:0/3357773339 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e800dd10 con 0x7f93f415ad70 2023-11-24T06:08:40.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.471+0000 7f93f37fe700 1 -- 172.21.15.91:0/3357773339 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e800de70 con 0x7f93f415ad70 2023-11-24T06:08:40.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.471+0000 7f93f37fe700 1 -- 172.21.15.91:0/3357773339 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e8004250 con 0x7f93f415ad70 2023-11-24T06:08:40.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.471+0000 7f93faa8a700 1 -- 172.21.15.91:0/3357773339 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93f415ad70 msgr2=0x7f93f40b5e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.471+0000 7f93faa8a700 1 --2- 172.21.15.91:0/3357773339 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93f415ad70 0x7f93f40b5e40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f93e800bbc0 tx=0x7f93e800bed0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.471+0000 7f93faa8a700 1 -- 172.21.15.91:0/3357773339 shutdown_connections 2023-11-24T06:08:40.479 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.471+0000 7f93faa8a700 1 --2- 172.21.15.91:0/3357773339 >> v2:172.21.15.94:3300/0 conn(0x7f93f40adda0 0x7f93f406b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.479 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.471+0000 7f93faa8a700 1 --2- 172.21.15.91:0/3357773339 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93f415ad70 0x7f93f40b5e40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.479 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.471+0000 7f93faa8a700 1 -- 172.21.15.91:0/3357773339 >> 172.21.15.91:0/3357773339 conn(0x7f93f40b5070 msgr2=0x7f93f40b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:40.479 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.472+0000 7f93faa8a700 1 -- 172.21.15.91:0/3357773339 shutdown_connections 2023-11-24T06:08:40.479 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.472+0000 7f93faa8a700 1 -- 172.21.15.91:0/3357773339 wait complete. 2023-11-24T06:08:40.479 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.473+0000 7f93faa8a700 1 Processor -- start 2023-11-24T06:08:40.479 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.473+0000 7f93faa8a700 1 -- start start 2023-11-24T06:08:40.482 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93faa8a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f40adda0 0x7f93f414a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.482 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93faa8a700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93f40b03b0 0x7f93f415b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.482 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93faa8a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f93f415ad70 0x7f93f415d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.482 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93faa8a700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f414a5e0 con 0x7f93f40b03b0 2023-11-24T06:08:40.482 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93faa8a700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f4153b90 con 0x7f93f40adda0 2023-11-24T06:08:40.482 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93faa8a700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f415c250 con 0x7f93f415ad70 2023-11-24T06:08:40.482 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93f3fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93f40b03b0 0x7f93f415b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.482 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93f3fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93f40b03b0 0x7f93f415b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59062/0 (socket says 172.21.15.91:59062) 2023-11-24T06:08:40.482 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93f3fff700 1 -- 172.21.15.91:0/4178776632 learned_addr learned my addr 172.21.15.91:0/4178776632 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:40.482 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93f8826700 1 --2- 172.21.15.91:0/4178776632 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f40adda0 0x7f93f414a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.482 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93f9027700 1 --2- 172.21.15.91:0/4178776632 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f93f415ad70 0x7f93f415d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93f3fff700 1 -- 172.21.15.91:0/4178776632 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f93f415ad70 msgr2=0x7f93f415d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93f3fff700 1 --2- 172.21.15.91:0/4178776632 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f93f415ad70 0x7f93f415d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93f3fff700 1 -- 172.21.15.91:0/4178776632 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f40adda0 msgr2=0x7f93f414a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93f3fff700 1 --2- 172.21.15.91:0/4178776632 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f40adda0 0x7f93f414a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93f3fff700 1 -- 172.21.15.91:0/4178776632 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e800b7f0 con 0x7f93f40b03b0 2023-11-24T06:08:40.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93f8826700 1 --2- 172.21.15.91:0/4178776632 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f40adda0 0x7f93f414a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:40.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93f3fff700 1 --2- 172.21.15.91:0/4178776632 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93f40b03b0 0x7f93f415b4f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f93e000c4d0 tx=0x7f93e000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:40.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93f17fa700 1 -- 172.21.15.91:0/4178776632 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0011790 con 0x7f93f40b03b0 2023-11-24T06:08:40.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93f17fa700 1 -- 172.21.15.91:0/4178776632 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e00118f0 con 0x7f93f40b03b0 2023-11-24T06:08:40.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93faa8a700 1 -- 172.21.15.91:0/4178776632 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f415bf70 con 0x7f93f40b03b0 2023-11-24T06:08:40.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93f17fa700 1 -- 172.21.15.91:0/4178776632 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e0011ca0 con 0x7f93f40b03b0 2023-11-24T06:08:40.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.474+0000 7f93faa8a700 1 -- 172.21.15.91:0/4178776632 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f414c5c0 con 0x7f93f40b03b0 2023-11-24T06:08:40.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.475+0000 7f93faa8a700 1 -- 172.21.15.91:0/4178776632 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f414c140 con 0x7f93f40b03b0 2023-11-24T06:08:40.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.476+0000 7f93f17fa700 1 -- 172.21.15.91:0/4178776632 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f93e001c460 con 0x7f93f40b03b0 2023-11-24T06:08:40.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.476+0000 7f93f17fa700 1 --2- 172.21.15.91:0/4178776632 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93d806aed0 0x7f93d806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.476+0000 7f93f17fa700 1 -- 172.21.15.91:0/4178776632 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93e0089cc0 con 0x7f93f40b03b0 2023-11-24T06:08:40.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.477+0000 7f93f8826700 1 --2- 172.21.15.91:0/4178776632 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93d806aed0 0x7f93d806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.477+0000 7f93f8826700 1 --2- 172.21.15.91:0/4178776632 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93d806aed0 0x7f93d806d390 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f93e800b340 tx=0x7f93e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:40.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.480+0000 7f93f17fa700 1 -- 172.21.15.91:0/4178776632 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93e0054e50 con 0x7f93f40b03b0 2023-11-24T06:08:40.674 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.672+0000 7f93faa8a700 1 -- 172.21.15.91:0/4178776632 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f93f414c140 con 0x7f93f40b03b0 2023-11-24T06:08:40.675 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.672+0000 7f93f17fa700 1 -- 172.21.15.91:0/4178776632 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f93f414c140 con 0x7f93f40b03b0 2023-11-24T06:08:40.675 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 64 2023-11-24T06:08:40.678 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.674+0000 7f93faa8a700 1 -- 172.21.15.91:0/4178776632 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93d806aed0 msgr2=0x7f93d806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.678 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.674+0000 7f93faa8a700 1 --2- 172.21.15.91:0/4178776632 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93d806aed0 0x7f93d806d390 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f93e800b340 tx=0x7f93e8008040 comp rx=0 tx=0).stop 2023-11-24T06:08:40.678 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.674+0000 7f93faa8a700 1 -- 172.21.15.91:0/4178776632 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93f40b03b0 msgr2=0x7f93f415b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.678 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.674+0000 7f93faa8a700 1 --2- 172.21.15.91:0/4178776632 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93f40b03b0 0x7f93f415b4f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f93e000c4d0 tx=0x7f93e000c890 comp rx=0 tx=0).stop 2023-11-24T06:08:40.678 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.675+0000 7f93faa8a700 1 -- 172.21.15.91:0/4178776632 shutdown_connections 2023-11-24T06:08:40.678 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.675+0000 7f93faa8a700 1 --2- 172.21.15.91:0/4178776632 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93d806aed0 0x7f93d806d390 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.678 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.675+0000 7f93faa8a700 1 --2- 172.21.15.91:0/4178776632 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93f40adda0 0x7f93f414a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.678 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.675+0000 7f93faa8a700 1 --2- 172.21.15.91:0/4178776632 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f93f415ad70 0x7f93f415d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.678 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.675+0000 7f93faa8a700 1 --2- 172.21.15.91:0/4178776632 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93f40b03b0 0x7f93f415b4f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.678 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.675+0000 7f93faa8a700 1 -- 172.21.15.91:0/4178776632 >> 172.21.15.91:0/4178776632 conn(0x7f93f40b5070 msgr2=0x7f93f40b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:40.679 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.675+0000 7f93faa8a700 1 -- 172.21.15.91:0/4178776632 shutdown_connections 2023-11-24T06:08:40.679 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.675+0000 7f93faa8a700 1 -- 172.21.15.91:0/4178776632 wait complete. 2023-11-24T06:08:40.689 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T06:08:40.811 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.808+0000 7f1fe76c5700 1 Processor -- start 2023-11-24T06:08:40.811 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.808+0000 7f1fe76c5700 1 -- start start 2023-11-24T06:08:40.814 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fe76c5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1fe015bde0 0x7f1fe015c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fe76c5700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00d0a90 con 0x7f1fe00ce5c0 2023-11-24T06:08:40.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fe76c5700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe006ee10 con 0x7f1fe00cf2b0 2023-11-24T06:08:40.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fe76c5700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe006f0f0 con 0x7f1fe015bde0 2023-11-24T06:08:40.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fe5c62700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f1fe00ce5c0 0x7f1fe006fbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54006/0 (socket says 172.21.15.91:54006) 2023-11-24T06:08:40.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fe5c62700 1 -- 172.21.15.91:0/3806749310 learned_addr learned my addr 172.21.15.91:0/3806749310 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:40.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fd7fff700 1 -- 172.21.15.91:0/3806749310 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632556615 0 0) 0x7f1fe00d0a90 con 0x7f1fe00ce5c0 2023-11-24T06:08:40.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fe5461700 1 --2- 172.21.15.91:0/3806749310 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1fe015bde0 0x7f1fe015c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fd7fff700 1 -- 172.21.15.91:0/3806749310 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fcc0035b0 con 0x7f1fe00ce5c0 2023-11-24T06:08:40.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fd7fff700 1 -- 172.21.15.91:0/3806749310 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1097726638 0 0) 0x7f1fcc0035b0 con 0x7f1fe00ce5c0 2023-11-24T06:08:40.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fd7fff700 1 -- 172.21.15.91:0/3806749310 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1fe015bde0 msgr2=0x7f1fe015c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:40.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fd7fff700 1 --2- 172.21.15.91:0/3806749310 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1fe015bde0 0x7f1fe015c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fd7fff700 1 -- 172.21.15.91:0/3806749310 >> v1:172.21.15.94:6789/0 conn(0x7f1fe00cf2b0 legacy=0x7f1fe00d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fd7fff700 1 -- 172.21.15.91:0/3806749310 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe006f3d0 con 0x7f1fe00ce5c0 2023-11-24T06:08:40.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fd7fff700 1 -- 172.21.15.91:0/3806749310 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f1fd80019f0 con 0x7f1fe00ce5c0 2023-11-24T06:08:40.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fd7fff700 1 -- 172.21.15.91:0/3806749310 >> v1:172.21.15.91:6789/0 conn(0x7f1fe00ce5c0 legacy=0x7f1fe006fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fd7fff700 1 --2- 172.21.15.91:0/3806749310 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fcc003a30 0x7f1fcc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.809+0000 7f1fd7fff700 1 --2- 172.21.15.91:0/3806749310 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1fcc004850 0x7f1fcc004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fd7fff700 1 --2- 172.21.15.91:0/3806749310 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1fe015bde0 0x7f1fcc005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fd7fff700 1 -- 172.21.15.91:0/3806749310 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe006f0f0 con 0x7f1fcc004850 2023-11-24T06:08:40.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fd7fff700 1 -- 172.21.15.91:0/3806749310 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc00bb40 con 0x7f1fcc003a30 2023-11-24T06:08:40.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fd7fff700 1 -- 172.21.15.91:0/3806749310 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc00bd00 con 0x7f1fe015bde0 2023-11-24T06:08:40.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fe4c60700 1 --2- 172.21.15.91:0/3806749310 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1fcc004850 0x7f1fcc004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fe5461700 1 --2- 172.21.15.91:0/3806749310 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fcc003a30 0x7f1fcc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.817 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fe5c62700 1 --2- 172.21.15.91:0/3806749310 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1fe015bde0 0x7f1fcc005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.817 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fe4c60700 1 -- 172.21.15.91:0/3806749310 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1fe015bde0 msgr2=0x7f1fcc005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:40.817 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fe4c60700 1 --2- 172.21.15.91:0/3806749310 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1fe015bde0 0x7f1fcc005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.817 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fe4c60700 1 -- 172.21.15.91:0/3806749310 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fcc003a30 msgr2=0x7f1fcc003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:40.817 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fe4c60700 1 --2- 172.21.15.91:0/3806749310 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fcc003a30 0x7f1fcc003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.817 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fe4c60700 1 -- 172.21.15.91:0/3806749310 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc0035b0 con 0x7f1fcc004850 2023-11-24T06:08:40.817 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fe5c62700 1 --2- 172.21.15.91:0/3806749310 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1fe015bde0 0x7f1fcc005250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:40.817 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fe4c60700 1 --2- 172.21.15.91:0/3806749310 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1fcc004850 0x7f1fcc004cd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00cf710 tx=0x7f1fcc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8919ee39e668e2f1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:40.817 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fd7fff700 1 -- 172.21.15.91:0/3806749310 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd00052d0 con 0x7f1fcc004850 2023-11-24T06:08:40.817 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fd7fff700 1 -- 172.21.15.91:0/3806749310 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd0005470 con 0x7f1fcc004850 2023-11-24T06:08:40.817 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.810+0000 7f1fd7fff700 1 -- 172.21.15.91:0/3806749310 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd00038c0 con 0x7f1fcc004850 2023-11-24T06:08:40.817 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.811+0000 7f1fe76c5700 1 -- 172.21.15.91:0/3806749310 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1fcc004850 msgr2=0x7f1fcc004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.817 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.811+0000 7f1fe76c5700 1 --2- 172.21.15.91:0/3806749310 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1fcc004850 0x7f1fcc004cd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00cf710 tx=0x7f1fcc005880 comp rx=0 tx=0).stop 2023-11-24T06:08:40.818 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.811+0000 7f1fe5c62700 1 -- 172.21.15.91:0/3806749310 reap_dead start 2023-11-24T06:08:40.818 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.811+0000 7f1fe76c5700 1 -- 172.21.15.91:0/3806749310 shutdown_connections 2023-11-24T06:08:40.818 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.811+0000 7f1fe76c5700 1 -- 172.21.15.91:0/3806749310 >> 172.21.15.91:0/3806749310 conn(0x7f1fe00b3720 msgr2=0x7f1fe00b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:40.818 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.811+0000 7f1fe76c5700 1 -- 172.21.15.91:0/3806749310 shutdown_connections 2023-11-24T06:08:40.818 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.811+0000 7f1fe76c5700 1 -- 172.21.15.91:0/3806749310 wait complete. 2023-11-24T06:08:40.818 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.812+0000 7f1fe76c5700 1 Processor -- start 2023-11-24T06:08:40.818 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.812+0000 7f1fe76c5700 1 -- start start 2023-11-24T06:08:40.820 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe76c5700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1fcc004850 0x7f1fe01f20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe76c5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1fe00b6cb0 0x7f1fe00c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe76c5700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe014fcb0 0x7f1fe0150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe76c5700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe0150680 con 0x7f1fcc004850 2023-11-24T06:08:40.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe76c5700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00b6820 con 0x7f1fe014fcb0 2023-11-24T06:08:40.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe76c5700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe01f2610 con 0x7f1fe00b6cb0 2023-11-24T06:08:40.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe5461700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1fcc004850 0x7f1fe01f20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe5461700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1fcc004850 0x7f1fe01f20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59080/0 (socket says 172.21.15.91:59080) 2023-11-24T06:08:40.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe5461700 1 -- 172.21.15.91:0/3455372426 learned_addr learned my addr 172.21.15.91:0/3455372426 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:40.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe4c60700 1 --2- 172.21.15.91:0/3455372426 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1fe00b6cb0 0x7f1fe00c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe5c62700 1 --2- 172.21.15.91:0/3455372426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe014fcb0 0x7f1fe0150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe5461700 1 -- 172.21.15.91:0/3455372426 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1fe00b6cb0 msgr2=0x7f1fe00c9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.821 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe5461700 1 --2- 172.21.15.91:0/3455372426 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1fe00b6cb0 0x7f1fe00c9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe5461700 1 -- 172.21.15.91:0/3455372426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe014fcb0 msgr2=0x7f1fe0150140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:40.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe5461700 1 --2- 172.21.15.91:0/3455372426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe014fcb0 0x7f1fe0150140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:40.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe5461700 1 -- 172.21.15.91:0/3455372426 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe00b5c10 con 0x7f1fcc004850 2023-11-24T06:08:40.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe5c62700 1 --2- 172.21.15.91:0/3455372426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe014fcb0 0x7f1fe0150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:40.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe5461700 1 --2- 172.21.15.91:0/3455372426 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1fcc004850 0x7f1fe01f20d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc011ad0 tx=0x7f1fdc011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:40.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fd5ffb700 1 -- 172.21.15.91:0/3455372426 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc00f8e0 con 0x7f1fcc004850 2023-11-24T06:08:40.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fd5ffb700 1 -- 172.21.15.91:0/3455372426 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc00fa40 con 0x7f1fcc004850 2023-11-24T06:08:40.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fe76c5700 1 -- 172.21.15.91:0/3455372426 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe00b5ef0 con 0x7f1fcc004850 2023-11-24T06:08:40.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.813+0000 7f1fd5ffb700 1 -- 172.21.15.91:0/3455372426 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc00fdf0 con 0x7f1fcc004850 2023-11-24T06:08:40.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.814+0000 7f1fe76c5700 1 -- 172.21.15.91:0/3455372426 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe00b64b0 con 0x7f1fcc004850 2023-11-24T06:08:40.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.814+0000 7f1fe76c5700 1 -- 172.21.15.91:0/3455372426 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe0068670 con 0x7f1fcc004850 2023-11-24T06:08:40.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.815+0000 7f1fd5ffb700 1 -- 172.21.15.91:0/3455372426 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1fdc014440 con 0x7f1fcc004850 2023-11-24T06:08:40.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.815+0000 7f1fd5ffb700 1 --2- 172.21.15.91:0/3455372426 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1fc406aed0 0x7f1fc406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:40.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.815+0000 7f1fd5ffb700 1 -- 172.21.15.91:0/3455372426 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fdc089d40 con 0x7f1fcc004850 2023-11-24T06:08:40.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.815+0000 7f1fe4c60700 1 --2- 172.21.15.91:0/3455372426 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1fc406aed0 0x7f1fc406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:40.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.816+0000 7f1fe4c60700 1 --2- 172.21.15.91:0/3455372426 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1fc406aed0 0x7f1fc406d390 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00cf710 tx=0x7f1fd0009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:40.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:40.820+0000 7f1fd5ffb700 1 -- 172.21.15.91:0/3455372426 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1fdc054e10 con 0x7f1fcc004850 2023-11-24T06:08:41.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.027+0000 7f1fe76c5700 1 -- 172.21.15.91:0/3455372426 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1fe0074580 con 0x7f1fcc004850 2023-11-24T06:08:41.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.027+0000 7f1fd5ffb700 1 -- 172.21.15.91:0/3455372426 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1fdc058430 con 0x7f1fcc004850 2023-11-24T06:08:41.032 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.029+0000 7f1fe76c5700 1 -- 172.21.15.91:0/3455372426 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1fc406aed0 msgr2=0x7f1fc406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:41.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.029+0000 7f1fe76c5700 1 --2- 172.21.15.91:0/3455372426 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1fc406aed0 0x7f1fc406d390 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00cf710 tx=0x7f1fd0009030 comp rx=0 tx=0).stop 2023-11-24T06:08:41.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.029+0000 7f1fe76c5700 1 -- 172.21.15.91:0/3455372426 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1fcc004850 msgr2=0x7f1fe01f20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:41.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.029+0000 7f1fe76c5700 1 --2- 172.21.15.91:0/3455372426 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1fcc004850 0x7f1fe01f20d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc011ad0 tx=0x7f1fdc011de0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.029+0000 7f1fe76c5700 1 -- 172.21.15.91:0/3455372426 shutdown_connections 2023-11-24T06:08:41.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.029+0000 7f1fe76c5700 1 --2- 172.21.15.91:0/3455372426 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1fc406aed0 0x7f1fc406d390 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.029+0000 7f1fe76c5700 1 --2- 172.21.15.91:0/3455372426 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1fe014fcb0 0x7f1fe0150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.029+0000 7f1fe76c5700 1 --2- 172.21.15.91:0/3455372426 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1fcc004850 0x7f1fe01f20d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.029+0000 7f1fe76c5700 1 --2- 172.21.15.91:0/3455372426 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1fe00b6cb0 0x7f1fe00c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.029+0000 7f1fe76c5700 1 -- 172.21.15.91:0/3455372426 >> 172.21.15.91:0/3455372426 conn(0x7f1fe00b3720 msgr2=0x7f1fe00cc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:41.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.029+0000 7f1fe76c5700 1 -- 172.21.15.91:0/3455372426 shutdown_connections 2023-11-24T06:08:41.033 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.029+0000 7f1fe76c5700 1 -- 172.21.15.91:0/3455372426 wait complete. 2023-11-24T06:08:41.034 INFO:teuthology.orchestra.run.smithi091.stderr:listed 0 entries 2023-11-24T06:08:41.044 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2023-11-24T06:08:41.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:41.044 DEBUG:teuthology.orchestra.run.smithi091:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2023-11-24T06:08:41.089 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi091.front.sepia.ceph.com 2023-11-24T06:08:41.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:41.090 DEBUG:teuthology.orchestra.run.smithi091:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-11-24T06:08:41.106 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:08:41.106 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:08:41.127 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:08:41.128 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:08:41.189 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:08:41.190 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:08:41.251 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:41.370 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.368+0000 7f55b3593700 1 Processor -- start 2023-11-24T06:08:41.371 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.368+0000 7f55b3593700 1 -- start start 2023-11-24T06:08:41.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.369+0000 7f55b3593700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f55ac143ae0 0x7f55ac0b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:41.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.369+0000 7f55b3593700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f55ac0722c0 0x7f55ac06e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:41.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.369+0000 7f55b3593700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55ac143ec0 con 0x7f55ac1440f0 2023-11-24T06:08:41.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.369+0000 7f55b3593700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55ac0b40a0 con 0x7f55ac0722c0 2023-11-24T06:08:41.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.369+0000 7f55b3593700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55ac0b3ae0 con 0x7f55ac143ae0 2023-11-24T06:08:41.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.369+0000 7f55b1b30700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f55ac0722c0 0x7f55ac06e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:41.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.369+0000 7f55b0b2e700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f55ac1440f0 0x7f55ac071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54022/0 (socket says 172.21.15.91:54022) 2023-11-24T06:08:41.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.369+0000 7f55b1b30700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f55ac0722c0 0x7f55ac06e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59092/0 (socket says 172.21.15.91:59092) 2023-11-24T06:08:41.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.369+0000 7f55b0b2e700 1 -- 172.21.15.91:0/3900033057 learned_addr learned my addr 172.21.15.91:0/3900033057 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:41.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.369+0000 7f55b132f700 1 --2- 172.21.15.91:0/3900033057 >> v2:172.21.15.94:3300/0 conn(0x7f55ac143ae0 0x7f55ac0b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:41.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.369+0000 7f55b1b30700 1 -- 172.21.15.91:0/3900033057 >> v2:172.21.15.94:3300/0 conn(0x7f55ac143ae0 msgr2=0x7f55ac0b51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:41.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.369+0000 7f55b1b30700 1 --2- 172.21.15.91:0/3900033057 >> v2:172.21.15.94:3300/0 conn(0x7f55ac143ae0 0x7f55ac0b51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.369+0000 7f55b1b30700 1 -- 172.21.15.91:0/3900033057 >> v1:172.21.15.91:6789/0 conn(0x7f55ac1440f0 legacy=0x7f55ac071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:41.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.369+0000 7f55a3fff700 1 -- 172.21.15.91:0/3900033057 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567478520 0 0) 0x7f55ac143ec0 con 0x7f55ac1440f0 2023-11-24T06:08:41.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.369+0000 7f55b1b30700 1 -- 172.21.15.91:0/3900033057 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ac0b3dc0 con 0x7f55ac0722c0 2023-11-24T06:08:41.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.369+0000 7f55b1b30700 1 --2- 172.21.15.91:0/3900033057 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55ac0722c0 0x7f55ac06e510 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f55ac06d2e0 tx=0x7f55a400d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc7b1d68bc00f3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:41.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.370+0000 7f55a3fff700 1 -- 172.21.15.91:0/3900033057 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a4010070 con 0x7f55ac0722c0 2023-11-24T06:08:41.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.370+0000 7f55a3fff700 1 -- 172.21.15.91:0/3900033057 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a400b2e0 con 0x7f55ac0722c0 2023-11-24T06:08:41.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.370+0000 7f55a3fff700 1 -- 172.21.15.91:0/3900033057 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a400dc10 con 0x7f55ac0722c0 2023-11-24T06:08:41.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.370+0000 7f55b3593700 1 -- 172.21.15.91:0/3900033057 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55ac0722c0 msgr2=0x7f55ac06e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:41.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.370+0000 7f55b3593700 1 --2- 172.21.15.91:0/3900033057 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55ac0722c0 0x7f55ac06e510 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f55ac06d2e0 tx=0x7f55a400d330 comp rx=0 tx=0).stop 2023-11-24T06:08:41.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.370+0000 7f55b3593700 1 -- 172.21.15.91:0/3900033057 shutdown_connections 2023-11-24T06:08:41.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.370+0000 7f55b3593700 1 --2- 172.21.15.91:0/3900033057 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55ac0722c0 0x7f55ac06e510 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.370+0000 7f55b3593700 1 --2- 172.21.15.91:0/3900033057 >> v2:172.21.15.94:3300/0 conn(0x7f55ac143ae0 0x7f55ac0b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.370+0000 7f55b3593700 1 -- 172.21.15.91:0/3900033057 >> 172.21.15.91:0/3900033057 conn(0x7f55ac0b2740 msgr2=0x7f55ac0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:41.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.370+0000 7f55b3593700 1 -- 172.21.15.91:0/3900033057 shutdown_connections 2023-11-24T06:08:41.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.370+0000 7f55b3593700 1 -- 172.21.15.91:0/3900033057 wait complete. 2023-11-24T06:08:41.380 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.371+0000 7f55b3593700 1 Processor -- start 2023-11-24T06:08:41.380 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.371+0000 7f55b3593700 1 -- start start 2023-11-24T06:08:41.380 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.372+0000 7f55b3593700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55ac0722c0 0x7f55ac15e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:41.380 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.372+0000 7f55b3593700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55ac143ae0 0x7f55ac0cb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:41.380 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.372+0000 7f55b3593700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55ac1440f0 0x7f55ac0cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:41.380 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.372+0000 7f55b3593700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac0cccf0 con 0x7f55ac0722c0 2023-11-24T06:08:41.380 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.372+0000 7f55b3593700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac0cc0e0 con 0x7f55ac143ae0 2023-11-24T06:08:41.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.372+0000 7f55b3593700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac0cc3c0 con 0x7f55ac1440f0 2023-11-24T06:08:41.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.372+0000 7f55b132f700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55ac0722c0 0x7f55ac15e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:41.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.372+0000 7f55b132f700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55ac0722c0 0x7f55ac15e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59094/0 (socket says 172.21.15.91:59094) 2023-11-24T06:08:41.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.372+0000 7f55b132f700 1 -- 172.21.15.91:0/727888348 learned_addr learned my addr 172.21.15.91:0/727888348 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:41.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.372+0000 7f55b132f700 1 -- 172.21.15.91:0/727888348 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55ac1440f0 msgr2=0x7f55ac0cba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:41.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.372+0000 7f55b132f700 1 --2- 172.21.15.91:0/727888348 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55ac1440f0 0x7f55ac0cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.382 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.372+0000 7f55b132f700 1 -- 172.21.15.91:0/727888348 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55ac143ae0 msgr2=0x7f55ac0cb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:41.382 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.372+0000 7f55b132f700 1 --2- 172.21.15.91:0/727888348 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55ac143ae0 0x7f55ac0cb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.382 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.372+0000 7f55b132f700 1 -- 172.21.15.91:0/727888348 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a400bc80 con 0x7f55ac0722c0 2023-11-24T06:08:41.382 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.373+0000 7f55b132f700 1 --2- 172.21.15.91:0/727888348 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55ac0722c0 0x7f55ac15e5c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f55a800e9b0 tx=0x7f55a800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:41.382 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.373+0000 7f55a1ffb700 1 -- 172.21.15.91:0/727888348 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a8009710 con 0x7f55ac0722c0 2023-11-24T06:08:41.382 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.373+0000 7f55a1ffb700 1 -- 172.21.15.91:0/727888348 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a800c850 con 0x7f55ac0722c0 2023-11-24T06:08:41.382 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.373+0000 7f55b3593700 1 -- 172.21.15.91:0/727888348 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ac0cc6a0 con 0x7f55ac0722c0 2023-11-24T06:08:41.382 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.373+0000 7f55a1ffb700 1 -- 172.21.15.91:0/727888348 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a800cbc0 con 0x7f55ac0722c0 2023-11-24T06:08:41.383 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.373+0000 7f55b3593700 1 -- 172.21.15.91:0/727888348 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55ac15a7e0 con 0x7f55ac0722c0 2023-11-24T06:08:41.383 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.374+0000 7f55b3593700 1 -- 172.21.15.91:0/727888348 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ac068670 con 0x7f55ac0722c0 2023-11-24T06:08:41.383 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.374+0000 7f55a1ffb700 1 -- 172.21.15.91:0/727888348 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f55a800cd20 con 0x7f55ac0722c0 2023-11-24T06:08:41.383 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.375+0000 7f55a1ffb700 1 --2- 172.21.15.91:0/727888348 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f559006aed0 0x7f559006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:41.383 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.375+0000 7f55a1ffb700 1 -- 172.21.15.91:0/727888348 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55a8089640 con 0x7f55ac0722c0 2023-11-24T06:08:41.383 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.375+0000 7f55b0b2e700 1 --2- 172.21.15.91:0/727888348 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f559006aed0 0x7f559006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:41.384 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.375+0000 7f55b0b2e700 1 --2- 172.21.15.91:0/727888348 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f559006aed0 0x7f559006d390 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f55ac141740 tx=0x7f559c0079f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:41.384 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.379+0000 7f55a1ffb700 1 -- 172.21.15.91:0/727888348 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f55a8054780 con 0x7f55ac0722c0 2023-11-24T06:08:41.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.577+0000 7f55b3593700 1 -- 172.21.15.91:0/727888348 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55ac0b5d20 con 0x7f55ac0722c0 2023-11-24T06:08:41.581 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.578+0000 7f55a1ffb700 1 -- 172.21.15.91:0/727888348 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7f55a8057da0 con 0x7f55ac0722c0 2023-11-24T06:08:41.581 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:41.582 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":20,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:02.415612+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:41.583 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.580+0000 7f55b3593700 1 -- 172.21.15.91:0/727888348 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f559006aed0 msgr2=0x7f559006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:41.583 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.580+0000 7f55b3593700 1 --2- 172.21.15.91:0/727888348 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f559006aed0 0x7f559006d390 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f55ac141740 tx=0x7f559c0079f0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.580+0000 7f55b3593700 1 -- 172.21.15.91:0/727888348 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55ac0722c0 msgr2=0x7f55ac15e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:41.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.580+0000 7f55b3593700 1 --2- 172.21.15.91:0/727888348 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55ac0722c0 0x7f55ac15e5c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f55a800e9b0 tx=0x7f55a800ed70 comp rx=0 tx=0).stop 2023-11-24T06:08:41.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.580+0000 7f55b3593700 1 -- 172.21.15.91:0/727888348 shutdown_connections 2023-11-24T06:08:41.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.580+0000 7f55b3593700 1 --2- 172.21.15.91:0/727888348 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f559006aed0 0x7f559006d390 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.580+0000 7f55b3593700 1 --2- 172.21.15.91:0/727888348 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55ac1440f0 0x7f55ac0cba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.580+0000 7f55b3593700 1 --2- 172.21.15.91:0/727888348 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55ac0722c0 0x7f55ac15e5c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.580+0000 7f55b3593700 1 --2- 172.21.15.91:0/727888348 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55ac143ae0 0x7f55ac0cb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.580+0000 7f55b3593700 1 -- 172.21.15.91:0/727888348 >> 172.21.15.91:0/727888348 conn(0x7f55ac0b2740 msgr2=0x7f55ac0b41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:41.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.580+0000 7f55b3593700 1 -- 172.21.15.91:0/727888348 shutdown_connections 2023-11-24T06:08:41.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.581+0000 7f55b3593700 1 -- 172.21.15.91:0/727888348 wait complete. 2023-11-24T06:08:41.595 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:07:58.710106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:07:59.774819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:08:41.596 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:08:41.719 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.716+0000 7f803d49c700 1 Processor -- start 2023-11-24T06:08:41.719 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.716+0000 7f803d49c700 1 -- start start 2023-11-24T06:08:41.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.717+0000 7f803d49c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80380d2b50 0x7f80380cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:41.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.717+0000 7f803d49c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f80380d0740 0x7f80380d0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:41.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.717+0000 7f803d49c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80380ce9a0 con 0x7f803815e080 2023-11-24T06:08:41.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.717+0000 7f803d49c700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80380d14b0 con 0x7f80380d0740 2023-11-24T06:08:41.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.717+0000 7f803d49c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80380cec40 con 0x7f80380d2b50 2023-11-24T06:08:41.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.717+0000 7f80377fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f80380d0740 0x7f80380d0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:41.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.717+0000 7f80377fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f80380d0740 0x7f80380d0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59106/0 (socket says 172.21.15.91:59106) 2023-11-24T06:08:41.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.717+0000 7f80377fe700 1 -- 172.21.15.91:0/3276564908 learned_addr learned my addr 172.21.15.91:0/3276564908 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:41.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.717+0000 7f80367fc700 1 --2- 172.21.15.91:0/3276564908 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80380d2b50 0x7f80380cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:41.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.717+0000 7f80377fe700 1 -- 172.21.15.91:0/3276564908 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80380d2b50 msgr2=0x7f80380cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:41.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.717+0000 7f80377fe700 1 --2- 172.21.15.91:0/3276564908 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80380d2b50 0x7f80380cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.717+0000 7f80377fe700 1 -- 172.21.15.91:0/3276564908 >> v1:172.21.15.94:6789/0 conn(0x7f803815e080 legacy=0x7f80380d2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:41.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.717+0000 7f80377fe700 1 -- 172.21.15.91:0/3276564908 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80380cef20 con 0x7f80380d0740 2023-11-24T06:08:41.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.717+0000 7f80377fe700 1 --2- 172.21.15.91:0/3276564908 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f80380d0740 0x7f80380d0bf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8030009e50 tx=0x7f803000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94460ed035d75d68 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:41.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.717+0000 7f8035ffb700 1 -- 172.21.15.91:0/3276564908 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803000e070 con 0x7f80380d0740 2023-11-24T06:08:41.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.718+0000 7f8035ffb700 1 -- 172.21.15.91:0/3276564908 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803000b9c0 con 0x7f80380d0740 2023-11-24T06:08:41.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.718+0000 7f8035ffb700 1 -- 172.21.15.91:0/3276564908 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803000bd70 con 0x7f80380d0740 2023-11-24T06:08:41.723 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.718+0000 7f803d49c700 1 -- 172.21.15.91:0/3276564908 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f80380d0740 msgr2=0x7f80380d0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:41.723 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.718+0000 7f803d49c700 1 --2- 172.21.15.91:0/3276564908 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f80380d0740 0x7f80380d0bf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8030009e50 tx=0x7f803000b330 comp rx=0 tx=0).stop 2023-11-24T06:08:41.723 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.718+0000 7f803d49c700 1 -- 172.21.15.91:0/3276564908 shutdown_connections 2023-11-24T06:08:41.723 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.718+0000 7f803d49c700 1 --2- 172.21.15.91:0/3276564908 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f80380d0740 0x7f80380d0bf0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.723 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.718+0000 7f803d49c700 1 --2- 172.21.15.91:0/3276564908 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f80380d2b50 0x7f80380cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.723 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.718+0000 7f803d49c700 1 -- 172.21.15.91:0/3276564908 >> 172.21.15.91:0/3276564908 conn(0x7f803809d180 msgr2=0x7f80380d3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:41.723 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.718+0000 7f803d49c700 1 -- 172.21.15.91:0/3276564908 shutdown_connections 2023-11-24T06:08:41.723 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.718+0000 7f803d49c700 1 -- 172.21.15.91:0/3276564908 wait complete. 2023-11-24T06:08:41.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.719+0000 7f803d49c700 1 Processor -- start 2023-11-24T06:08:41.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.719+0000 7f803d49c700 1 -- start start 2023-11-24T06:08:41.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.720+0000 7f803d49c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80380d0740 0x7f80380b9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:41.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.720+0000 7f803d49c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f80380d2b50 0x7f80380b7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:41.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.720+0000 7f803d49c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f803815e080 0x7f8038160de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:41.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.720+0000 7f803d49c700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80380ce040 con 0x7f80380d2b50 2023-11-24T06:08:41.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.720+0000 7f803d49c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80380caf60 con 0x7f80380d0740 2023-11-24T06:08:41.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.720+0000 7f803d49c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80380cb240 con 0x7f803815e080 2023-11-24T06:08:41.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.720+0000 7f80367fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f80380d2b50 0x7f80380b7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:41.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.720+0000 7f80367fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f80380d2b50 0x7f80380b7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59110/0 (socket says 172.21.15.91:59110) 2023-11-24T06:08:41.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.720+0000 7f80367fc700 1 -- 172.21.15.91:0/4251712934 learned_addr learned my addr 172.21.15.91:0/4251712934 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:41.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.720+0000 7f80377fe700 1 --2- 172.21.15.91:0/4251712934 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f803815e080 0x7f8038160de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:41.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.721+0000 7f8036ffd700 1 --2- 172.21.15.91:0/4251712934 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80380d0740 0x7f80380b9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:41.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.721+0000 7f80367fc700 1 -- 172.21.15.91:0/4251712934 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f803815e080 msgr2=0x7f8038160de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:41.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.721+0000 7f80367fc700 1 --2- 172.21.15.91:0/4251712934 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f803815e080 0x7f8038160de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.721+0000 7f80367fc700 1 -- 172.21.15.91:0/4251712934 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80380d0740 msgr2=0x7f80380b9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:41.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.721+0000 7f80367fc700 1 --2- 172.21.15.91:0/4251712934 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80380d0740 0x7f80380b9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.721+0000 7f80367fc700 1 -- 172.21.15.91:0/4251712934 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8030009b30 con 0x7f80380d2b50 2023-11-24T06:08:41.728 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.721+0000 7f8036ffd700 1 --2- 172.21.15.91:0/4251712934 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80380d0740 0x7f80380b9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:41.728 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.721+0000 7f80367fc700 1 --2- 172.21.15.91:0/4251712934 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f80380d2b50 0x7f80380b7b00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f802800e840 tx=0x7f802800eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:41.728 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.721+0000 7f801f7fe700 1 -- 172.21.15.91:0/4251712934 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8028013610 con 0x7f80380d2b50 2023-11-24T06:08:41.728 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.721+0000 7f801f7fe700 1 -- 172.21.15.91:0/4251712934 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8028013770 con 0x7f80380d2b50 2023-11-24T06:08:41.728 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.721+0000 7f803d49c700 1 -- 172.21.15.91:0/4251712934 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80380b8040 con 0x7f80380d2b50 2023-11-24T06:08:41.728 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.721+0000 7f801f7fe700 1 -- 172.21.15.91:0/4251712934 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8028013b00 con 0x7f80380d2b50 2023-11-24T06:08:41.728 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.721+0000 7f803d49c700 1 -- 172.21.15.91:0/4251712934 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80380b85a0 con 0x7f80380d2b50 2023-11-24T06:08:41.728 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.722+0000 7f803d49c700 1 -- 172.21.15.91:0/4251712934 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8038075480 con 0x7f80380d2b50 2023-11-24T06:08:41.728 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.722+0000 7f801f7fe700 1 -- 172.21.15.91:0/4251712934 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8028013c60 con 0x7f80380d2b50 2023-11-24T06:08:41.728 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.723+0000 7f801f7fe700 1 --2- 172.21.15.91:0/4251712934 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f802406af00 0x7f802406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:41.728 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.723+0000 7f801f7fe700 1 -- 172.21.15.91:0/4251712934 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80280890e0 con 0x7f80380d2b50 2023-11-24T06:08:41.728 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.723+0000 7f8036ffd700 1 --2- 172.21.15.91:0/4251712934 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f802406af00 0x7f802406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:41.729 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.723+0000 7f8036ffd700 1 --2- 172.21.15.91:0/4251712934 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f802406af00 0x7f802406d3c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f802c000f60 tx=0x7f802c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:41.730 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.727+0000 7f801f7fe700 1 -- 172.21.15.91:0/4251712934 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8028054220 con 0x7f80380d2b50 2023-11-24T06:08:41.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.923+0000 7f803d49c700 1 -- 172.21.15.91:0/4251712934 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f80380cb380 con 0x7f80380d2b50 2023-11-24T06:08:41.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.923+0000 7f801f7fe700 1 -- 172.21.15.91:0/4251712934 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f80380cb380 con 0x7f80380d2b50 2023-11-24T06:08:41.926 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:08:41.929 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.925+0000 7f803d49c700 1 -- 172.21.15.91:0/4251712934 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f802406af00 msgr2=0x7f802406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:41.930 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.925+0000 7f803d49c700 1 --2- 172.21.15.91:0/4251712934 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f802406af00 0x7f802406d3c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f802c000f60 tx=0x7f802c009040 comp rx=0 tx=0).stop 2023-11-24T06:08:41.930 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.925+0000 7f803d49c700 1 -- 172.21.15.91:0/4251712934 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f80380d2b50 msgr2=0x7f80380b7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:41.930 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.925+0000 7f803d49c700 1 --2- 172.21.15.91:0/4251712934 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f80380d2b50 0x7f80380b7b00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f802800e840 tx=0x7f802800eb50 comp rx=0 tx=0).stop 2023-11-24T06:08:41.930 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.925+0000 7f803d49c700 1 -- 172.21.15.91:0/4251712934 shutdown_connections 2023-11-24T06:08:41.930 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.925+0000 7f803d49c700 1 --2- 172.21.15.91:0/4251712934 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f802406af00 0x7f802406d3c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.930 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.925+0000 7f803d49c700 1 --2- 172.21.15.91:0/4251712934 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f80380d0740 0x7f80380b9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.925+0000 7f803d49c700 1 --2- 172.21.15.91:0/4251712934 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f803815e080 0x7f8038160de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.925+0000 7f803d49c700 1 --2- 172.21.15.91:0/4251712934 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f80380d2b50 0x7f80380b7b00 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:41.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.925+0000 7f803d49c700 1 -- 172.21.15.91:0/4251712934 >> 172.21.15.91:0/4251712934 conn(0x7f803809d180 msgr2=0x7f803815edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:41.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.925+0000 7f803d49c700 1 -- 172.21.15.91:0/4251712934 shutdown_connections 2023-11-24T06:08:41.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:41.925+0000 7f803d49c700 1 -- 172.21.15.91:0/4251712934 wait complete. 2023-11-24T06:08:41.940 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:08:42.063 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.060+0000 7f56fffc6700 1 Processor -- start 2023-11-24T06:08:42.063 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.060+0000 7f56fffc6700 1 -- start start 2023-11-24T06:08:42.065 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.061+0000 7f56fffc6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f56f815e1e0 0x7f56f80b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.065 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.061+0000 7f56fffc6700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f56f80b5e40 0x7f56f815f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.066 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.061+0000 7f56fffc6700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f806bb80 con 0x7f56f806b400 2023-11-24T06:08:42.066 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.061+0000 7f56fffc6700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f806be20 con 0x7f56f80b5e40 2023-11-24T06:08:42.066 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.061+0000 7f56fffc6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f806c100 con 0x7f56f815e1e0 2023-11-24T06:08:42.066 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.061+0000 7f56fe563700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f56f80b5e40 0x7f56f815f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.066 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.061+0000 7f56fe563700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f56f80b5e40 0x7f56f815f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59120/0 (socket says 172.21.15.91:59120) 2023-11-24T06:08:42.066 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.061+0000 7f56fe563700 1 -- 172.21.15.91:0/1610031271 learned_addr learned my addr 172.21.15.91:0/1610031271 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:42.066 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.061+0000 7f56fe563700 1 -- 172.21.15.91:0/1610031271 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f56f815e1e0 msgr2=0x7f56f80b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:42.067 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.061+0000 7f56fdd62700 1 --2- 172.21.15.91:0/1610031271 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f56f815e1e0 0x7f56f80b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.067 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.061+0000 7f56fe563700 1 --2- 172.21.15.91:0/1610031271 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f56f815e1e0 0x7f56f80b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.067 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.061+0000 7f56fe563700 1 -- 172.21.15.91:0/1610031271 >> v1:172.21.15.94:6789/0 conn(0x7f56f806b400 legacy=0x7f56f80726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:42.067 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.061+0000 7f56fe563700 1 -- 172.21.15.91:0/1610031271 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f806c3e0 con 0x7f56f80b5e40 2023-11-24T06:08:42.067 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.061+0000 7f56fdd62700 1 --2- 172.21.15.91:0/1610031271 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f56f815e1e0 0x7f56f80b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:42.067 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.061+0000 7f56fe563700 1 --2- 172.21.15.91:0/1610031271 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f56f80b5e40 0x7f56f815f5d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f56f0009e50 tx=0x7f56f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd1a76f334b94a4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:42.067 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.062+0000 7f56fcd60700 1 -- 172.21.15.91:0/1610031271 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f000e070 con 0x7f56f80b5e40 2023-11-24T06:08:42.068 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.062+0000 7f56fcd60700 1 -- 172.21.15.91:0/1610031271 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f0003a20 con 0x7f56f80b5e40 2023-11-24T06:08:42.068 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.062+0000 7f56fcd60700 1 -- 172.21.15.91:0/1610031271 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f0003dd0 con 0x7f56f80b5e40 2023-11-24T06:08:42.068 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.062+0000 7f56fffc6700 1 -- 172.21.15.91:0/1610031271 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f56f80b5e40 msgr2=0x7f56f815f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:42.068 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.062+0000 7f56fffc6700 1 --2- 172.21.15.91:0/1610031271 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f56f80b5e40 0x7f56f815f5d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f56f0009e50 tx=0x7f56f000b330 comp rx=0 tx=0).stop 2023-11-24T06:08:42.068 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.062+0000 7f56fffc6700 1 -- 172.21.15.91:0/1610031271 shutdown_connections 2023-11-24T06:08:42.068 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.062+0000 7f56fffc6700 1 --2- 172.21.15.91:0/1610031271 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f56f80b5e40 0x7f56f815f5d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.069 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.062+0000 7f56fffc6700 1 --2- 172.21.15.91:0/1610031271 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f56f815e1e0 0x7f56f80b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.069 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.062+0000 7f56fffc6700 1 -- 172.21.15.91:0/1610031271 >> 172.21.15.91:0/1610031271 conn(0x7f56f80b5070 msgr2=0x7f56f80b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:42.069 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.062+0000 7f56fffc6700 1 -- 172.21.15.91:0/1610031271 shutdown_connections 2023-11-24T06:08:42.069 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.062+0000 7f56fffc6700 1 -- 172.21.15.91:0/1610031271 wait complete. 2023-11-24T06:08:42.071 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.063+0000 7f56fffc6700 1 Processor -- start 2023-11-24T06:08:42.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.063+0000 7f56fffc6700 1 -- start start 2023-11-24T06:08:42.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.064+0000 7f56fffc6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f56f806b400 0x7f56f8146060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.064+0000 7f56fffc6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56f80b5e40 0x7f56f8158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.064+0000 7f56fffc6700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f56f815e1e0 0x7f56f8159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.064+0000 7f56fffc6700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f8147310 con 0x7f56f815e1e0 2023-11-24T06:08:42.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.064+0000 7f56fffc6700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f81465a0 con 0x7f56f80b5e40 2023-11-24T06:08:42.072 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.064+0000 7f56fffc6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f80aeac0 con 0x7f56f806b400 2023-11-24T06:08:42.073 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.064+0000 7f56fe563700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f56f815e1e0 0x7f56f8159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.073 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.064+0000 7f56fe563700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f56f815e1e0 0x7f56f8159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59126/0 (socket says 172.21.15.91:59126) 2023-11-24T06:08:42.073 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.064+0000 7f56fe563700 1 -- 172.21.15.91:0/3676450432 learned_addr learned my addr 172.21.15.91:0/3676450432 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:42.073 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.064+0000 7f56fdd62700 1 --2- 172.21.15.91:0/3676450432 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f56f806b400 0x7f56f8146060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.073 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.064+0000 7f56fe563700 1 -- 172.21.15.91:0/3676450432 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f56f806b400 msgr2=0x7f56f8146060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:42.073 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.064+0000 7f56fe563700 1 --2- 172.21.15.91:0/3676450432 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f56f806b400 0x7f56f8146060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.073 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.064+0000 7f56fe563700 1 -- 172.21.15.91:0/3676450432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56f80b5e40 msgr2=0x7f56f8158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:42.074 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.064+0000 7f56fe563700 1 --2- 172.21.15.91:0/3676450432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56f80b5e40 0x7f56f8158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.074 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.064+0000 7f56fe563700 1 -- 172.21.15.91:0/3676450432 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f0009b30 con 0x7f56f815e1e0 2023-11-24T06:08:42.074 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.065+0000 7f56fe563700 1 --2- 172.21.15.91:0/3676450432 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f56f815e1e0 0x7f56f8159080 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f56f80737f0 tx=0x7f56f0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:42.074 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.065+0000 7f56ee7fc700 1 -- 172.21.15.91:0/3676450432 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f0004240 con 0x7f56f815e1e0 2023-11-24T06:08:42.074 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.065+0000 7f56ee7fc700 1 -- 172.21.15.91:0/3676450432 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f00043a0 con 0x7f56f815e1e0 2023-11-24T06:08:42.074 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.065+0000 7f56fffc6700 1 -- 172.21.15.91:0/3676450432 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f8146970 con 0x7f56f815e1e0 2023-11-24T06:08:42.074 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.065+0000 7f56ee7fc700 1 -- 172.21.15.91:0/3676450432 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f0004750 con 0x7f56f815e1e0 2023-11-24T06:08:42.075 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.065+0000 7f56fffc6700 1 -- 172.21.15.91:0/3676450432 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f8146f30 con 0x7f56f815e1e0 2023-11-24T06:08:42.075 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.066+0000 7f56fffc6700 1 -- 172.21.15.91:0/3676450432 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f8147450 con 0x7f56f815e1e0 2023-11-24T06:08:42.075 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.066+0000 7f56ee7fc700 1 -- 172.21.15.91:0/3676450432 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f56f0019070 con 0x7f56f815e1e0 2023-11-24T06:08:42.075 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.067+0000 7f56ee7fc700 1 --2- 172.21.15.91:0/3676450432 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f56dc06aed0 0x7f56dc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.075 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.067+0000 7f56ee7fc700 1 -- 172.21.15.91:0/3676450432 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f56f008a6a0 con 0x7f56f815e1e0 2023-11-24T06:08:42.075 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.067+0000 7f56fdd62700 1 --2- 172.21.15.91:0/3676450432 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f56dc06aed0 0x7f56dc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.075 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.067+0000 7f56fdd62700 1 --2- 172.21.15.91:0/3676450432 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f56dc06aed0 0x7f56dc06d390 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f56e8006fd0 tx=0x7f56e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:42.076 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.071+0000 7f56ee7fc700 1 -- 172.21.15.91:0/3676450432 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56f8147450 con 0x7f56f815e1e0 2023-11-24T06:08:42.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.259+0000 7f56fffc6700 1 -- 172.21.15.91:0/3676450432 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f56f8147450 con 0x7f56f815e1e0 2023-11-24T06:08:42.262 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.260+0000 7f56ee7fc700 1 -- 172.21.15.91:0/3676450432 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f56f8147450 con 0x7f56f815e1e0 2023-11-24T06:08:42.263 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 32 2023-11-24T06:08:42.265 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.262+0000 7f56fffc6700 1 -- 172.21.15.91:0/3676450432 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f56dc06aed0 msgr2=0x7f56dc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:42.265 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.262+0000 7f56fffc6700 1 --2- 172.21.15.91:0/3676450432 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f56dc06aed0 0x7f56dc06d390 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f56e8006fd0 tx=0x7f56e8008040 comp rx=0 tx=0).stop 2023-11-24T06:08:42.265 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.262+0000 7f56fffc6700 1 -- 172.21.15.91:0/3676450432 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f56f815e1e0 msgr2=0x7f56f8159080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:42.265 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.262+0000 7f56fffc6700 1 --2- 172.21.15.91:0/3676450432 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f56f815e1e0 0x7f56f8159080 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f56f80737f0 tx=0x7f56f0004ad0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.265 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.262+0000 7f56fffc6700 1 -- 172.21.15.91:0/3676450432 shutdown_connections 2023-11-24T06:08:42.265 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.262+0000 7f56fffc6700 1 --2- 172.21.15.91:0/3676450432 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f56dc06aed0 0x7f56dc06d390 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.266 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.262+0000 7f56fffc6700 1 --2- 172.21.15.91:0/3676450432 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f56f815e1e0 0x7f56f8159080 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.266 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.262+0000 7f56fffc6700 1 --2- 172.21.15.91:0/3676450432 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f56f806b400 0x7f56f8146060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.266 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.262+0000 7f56fffc6700 1 --2- 172.21.15.91:0/3676450432 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f56f80b5e40 0x7f56f8158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.266 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.262+0000 7f56fffc6700 1 -- 172.21.15.91:0/3676450432 >> 172.21.15.91:0/3676450432 conn(0x7f56f80b5070 msgr2=0x7f56f806d730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:42.266 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.262+0000 7f56fffc6700 1 -- 172.21.15.91:0/3676450432 shutdown_connections 2023-11-24T06:08:42.266 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.262+0000 7f56fffc6700 1 -- 172.21.15.91:0/3676450432 wait complete. 2023-11-24T06:08:42.277 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:08:42.401 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.398+0000 7fd4deaf0700 1 Processor -- start 2023-11-24T06:08:42.401 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.398+0000 7fd4deaf0700 1 -- start start 2023-11-24T06:08:42.403 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.399+0000 7fd4deaf0700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd4d806b620 0x7fd4d80b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.403 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.399+0000 7fd4deaf0700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4d806ee60 0x7fd4d80b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.403 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.399+0000 7fd4deaf0700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d8071f50 con 0x7fd4d80b5ab0 2023-11-24T06:08:42.403 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.399+0000 7fd4deaf0700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4d8072230 con 0x7fd4d806b620 2023-11-24T06:08:42.403 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.399+0000 7fd4deaf0700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d8072510 con 0x7fd4d806ee60 2023-11-24T06:08:42.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.399+0000 7fd4dc88c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd4d806b620 0x7fd4d80b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.399+0000 7fd4dc88c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd4d806b620 0x7fd4d80b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59130/0 (socket says 172.21.15.91:59130) 2023-11-24T06:08:42.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.399+0000 7fd4dc88c700 1 -- 172.21.15.91:0/388126301 learned_addr learned my addr 172.21.15.91:0/388126301 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:42.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.399+0000 7fd4dd08d700 1 --2- 172.21.15.91:0/388126301 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4d806ee60 0x7fd4d80b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.399+0000 7fd4dc88c700 1 -- 172.21.15.91:0/388126301 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4d806ee60 msgr2=0x7fd4d80b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:42.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.399+0000 7fd4dc88c700 1 --2- 172.21.15.91:0/388126301 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4d806ee60 0x7fd4d80b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.399+0000 7fd4dc88c700 1 -- 172.21.15.91:0/388126301 >> v1:172.21.15.94:6789/0 conn(0x7fd4d80b5ab0 legacy=0x7fd4d8161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:42.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.399+0000 7fd4dc88c700 1 -- 172.21.15.91:0/388126301 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d80727f0 con 0x7fd4d806b620 2023-11-24T06:08:42.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.399+0000 7fd4dc88c700 1 --2- 172.21.15.91:0/388126301 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4d806b620 0x7fd4d80b3a40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc0098f0 tx=0x7fd4cc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39ea46939fd6e17d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:42.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.399+0000 7fd4d77fe700 1 -- 172.21.15.91:0/388126301 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc00b8a0 con 0x7fd4d806b620 2023-11-24T06:08:42.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.400+0000 7fd4d77fe700 1 -- 172.21.15.91:0/388126301 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4cc00ba00 con 0x7fd4d806b620 2023-11-24T06:08:42.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.400+0000 7fd4d77fe700 1 -- 172.21.15.91:0/388126301 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc00bd90 con 0x7fd4d806b620 2023-11-24T06:08:42.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.400+0000 7fd4deaf0700 1 -- 172.21.15.91:0/388126301 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4d806b620 msgr2=0x7fd4d80b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:42.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.400+0000 7fd4deaf0700 1 --2- 172.21.15.91:0/388126301 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4d806b620 0x7fd4d80b3a40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc0098f0 tx=0x7fd4cc009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:42.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.400+0000 7fd4deaf0700 1 -- 172.21.15.91:0/388126301 shutdown_connections 2023-11-24T06:08:42.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.400+0000 7fd4deaf0700 1 --2- 172.21.15.91:0/388126301 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4d806ee60 0x7fd4d80b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.400+0000 7fd4deaf0700 1 --2- 172.21.15.91:0/388126301 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4d806b620 0x7fd4d80b3a40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.400+0000 7fd4deaf0700 1 -- 172.21.15.91:0/388126301 >> 172.21.15.91:0/388126301 conn(0x7fd4d80b2740 msgr2=0x7fd4d80b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:42.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.400+0000 7fd4deaf0700 1 -- 172.21.15.91:0/388126301 shutdown_connections 2023-11-24T06:08:42.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.400+0000 7fd4deaf0700 1 -- 172.21.15.91:0/388126301 wait complete. 2023-11-24T06:08:42.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.401+0000 7fd4deaf0700 1 Processor -- start 2023-11-24T06:08:42.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.401+0000 7fd4deaf0700 1 -- start start 2023-11-24T06:08:42.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.402+0000 7fd4deaf0700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4d806b620 0x7fd4d81f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.402+0000 7fd4deaf0700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4d806ee60 0x7fd4d80ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.402+0000 7fd4deaf0700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4d80b5ab0 0x7fd4d80adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.402+0000 7fd4deaf0700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d81f5120 con 0x7fd4d80b5ab0 2023-11-24T06:08:42.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.402+0000 7fd4deaf0700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d81f7a90 con 0x7fd4d806b620 2023-11-24T06:08:42.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.402+0000 7fd4deaf0700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d81f7d30 con 0x7fd4d806ee60 2023-11-24T06:08:42.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.402+0000 7fd4dd08d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4d80b5ab0 0x7fd4d80adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.408 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.402+0000 7fd4dd08d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4d80b5ab0 0x7fd4d80adda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59140/0 (socket says 172.21.15.91:59140) 2023-11-24T06:08:42.409 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.402+0000 7fd4dd08d700 1 -- 172.21.15.91:0/3071404553 learned_addr learned my addr 172.21.15.91:0/3071404553 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:42.409 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.402+0000 7fd4dc88c700 1 --2- 172.21.15.91:0/3071404553 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4d806b620 0x7fd4d81f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.409 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.402+0000 7fd4d7fff700 1 --2- 172.21.15.91:0/3071404553 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4d806ee60 0x7fd4d80ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.409 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.403+0000 7fd4dd08d700 1 -- 172.21.15.91:0/3071404553 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4d806ee60 msgr2=0x7fd4d80ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:42.409 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.403+0000 7fd4dd08d700 1 --2- 172.21.15.91:0/3071404553 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4d806ee60 0x7fd4d80ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.409 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.403+0000 7fd4dd08d700 1 -- 172.21.15.91:0/3071404553 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4d806b620 msgr2=0x7fd4d81f7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:42.409 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.403+0000 7fd4dd08d700 1 --2- 172.21.15.91:0/3071404553 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4d806b620 0x7fd4d81f7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.409 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.403+0000 7fd4dd08d700 1 -- 172.21.15.91:0/3071404553 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4cc009550 con 0x7fd4d80b5ab0 2023-11-24T06:08:42.409 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.403+0000 7fd4d7fff700 1 --2- 172.21.15.91:0/3071404553 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4d806ee60 0x7fd4d80ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:42.409 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.403+0000 7fd4dd08d700 1 --2- 172.21.15.91:0/3071404553 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4d80b5ab0 0x7fd4d80adda0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0007e30 tx=0x7fd4d000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:42.409 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.403+0000 7fd4d57fa700 1 -- 172.21.15.91:0/3071404553 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d000ede0 con 0x7fd4d80b5ab0 2023-11-24T06:08:42.409 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.403+0000 7fd4d57fa700 1 -- 172.21.15.91:0/3071404553 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d0014420 con 0x7fd4d80b5ab0 2023-11-24T06:08:42.410 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.403+0000 7fd4deaf0700 1 -- 172.21.15.91:0/3071404553 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d81f8010 con 0x7fd4d80b5ab0 2023-11-24T06:08:42.410 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.403+0000 7fd4d57fa700 1 -- 172.21.15.91:0/3071404553 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d00147d0 con 0x7fd4d80b5ab0 2023-11-24T06:08:42.410 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.403+0000 7fd4deaf0700 1 -- 172.21.15.91:0/3071404553 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d81f5330 con 0x7fd4d80b5ab0 2023-11-24T06:08:42.410 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.404+0000 7fd4deaf0700 1 -- 172.21.15.91:0/3071404553 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d81f54c0 con 0x7fd4d80b5ab0 2023-11-24T06:08:42.410 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.404+0000 7fd4d57fa700 1 -- 172.21.15.91:0/3071404553 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd4d0014930 con 0x7fd4d80b5ab0 2023-11-24T06:08:42.410 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.405+0000 7fd4d57fa700 1 --2- 172.21.15.91:0/3071404553 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd4bc06aed0 0x7fd4bc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.410 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.405+0000 7fd4d57fa700 1 -- 172.21.15.91:0/3071404553 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd4d0089df0 con 0x7fd4d80b5ab0 2023-11-24T06:08:42.410 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.405+0000 7fd4dc88c700 1 --2- 172.21.15.91:0/3071404553 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd4bc06aed0 0x7fd4bc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.410 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.405+0000 7fd4dc88c700 1 --2- 172.21.15.91:0/3071404553 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd4bc06aed0 0x7fd4bc06d390 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc009c40 tx=0x7fd4cc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:42.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.409+0000 7fd4d57fa700 1 -- 172.21.15.91:0/3071404553 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd4d0054e20 con 0x7fd4d80b5ab0 2023-11-24T06:08:42.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.598+0000 7fd4deaf0700 1 -- 172.21.15.91:0/3071404553 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd4d8075480 con 0x7fd4d80b5ab0 2023-11-24T06:08:42.601 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.598+0000 7fd4d57fa700 1 -- 172.21.15.91:0/3071404553 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd4d0058440 con 0x7fd4d80b5ab0 2023-11-24T06:08:42.601 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 64 2023-11-24T06:08:42.603 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.600+0000 7fd4deaf0700 1 -- 172.21.15.91:0/3071404553 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd4bc06aed0 msgr2=0x7fd4bc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:42.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.600+0000 7fd4deaf0700 1 --2- 172.21.15.91:0/3071404553 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd4bc06aed0 0x7fd4bc06d390 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc009c40 tx=0x7fd4cc008030 comp rx=0 tx=0).stop 2023-11-24T06:08:42.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.600+0000 7fd4deaf0700 1 -- 172.21.15.91:0/3071404553 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4d80b5ab0 msgr2=0x7fd4d80adda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:42.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.600+0000 7fd4deaf0700 1 --2- 172.21.15.91:0/3071404553 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4d80b5ab0 0x7fd4d80adda0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0007e30 tx=0x7fd4d000e3c0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.600+0000 7fd4deaf0700 1 -- 172.21.15.91:0/3071404553 shutdown_connections 2023-11-24T06:08:42.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.600+0000 7fd4deaf0700 1 --2- 172.21.15.91:0/3071404553 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd4bc06aed0 0x7fd4bc06d390 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.600+0000 7fd4deaf0700 1 --2- 172.21.15.91:0/3071404553 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd4d806b620 0x7fd4d81f7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.600+0000 7fd4deaf0700 1 --2- 172.21.15.91:0/3071404553 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4d80b5ab0 0x7fd4d80adda0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.600+0000 7fd4deaf0700 1 --2- 172.21.15.91:0/3071404553 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4d806ee60 0x7fd4d80ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.600+0000 7fd4deaf0700 1 -- 172.21.15.91:0/3071404553 >> 172.21.15.91:0/3071404553 conn(0x7fd4d80b2740 msgr2=0x7fd4d80b14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:42.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.600+0000 7fd4deaf0700 1 -- 172.21.15.91:0/3071404553 shutdown_connections 2023-11-24T06:08:42.604 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.601+0000 7fd4deaf0700 1 -- 172.21.15.91:0/3071404553 wait complete. 2023-11-24T06:08:42.615 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T06:08:42.735 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.732+0000 7f49b36c3700 1 Processor -- start 2023-11-24T06:08:42.735 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.733+0000 7f49b36c3700 1 -- start start 2023-11-24T06:08:42.738 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.733+0000 7f49b36c3700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f49ac06b400 0x7f49ac15e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.738 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.733+0000 7f49b36c3700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f49ac0b5e40 0x7f49ac15eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.738 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.733+0000 7f49b36c3700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac06bb80 con 0x7f49ac15e1e0 2023-11-24T06:08:42.738 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.733+0000 7f49b36c3700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ac06be20 con 0x7f49ac0b5e40 2023-11-24T06:08:42.738 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.733+0000 7f49b36c3700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ac06c100 con 0x7f49ac06b400 2023-11-24T06:08:42.738 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.734+0000 7f49b1c60700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f49ac0b5e40 0x7f49ac15eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.738 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.734+0000 7f49b1c60700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f49ac0b5e40 0x7f49ac15eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59148/0 (socket says 172.21.15.91:59148) 2023-11-24T06:08:42.738 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.734+0000 7f49b1c60700 1 -- 172.21.15.91:0/2343052236 learned_addr learned my addr 172.21.15.91:0/2343052236 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:42.738 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.734+0000 7f49b0c5e700 1 --2- 172.21.15.91:0/2343052236 >> v2:172.21.15.94:3300/0 conn(0x7f49ac06b400 0x7f49ac15e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.734+0000 7f49b1c60700 1 -- 172.21.15.91:0/2343052236 >> v2:172.21.15.94:3300/0 conn(0x7f49ac06b400 msgr2=0x7f49ac15e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:42.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.734+0000 7f49b1c60700 1 --2- 172.21.15.91:0/2343052236 >> v2:172.21.15.94:3300/0 conn(0x7f49ac06b400 0x7f49ac15e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.734+0000 7f49b1c60700 1 -- 172.21.15.91:0/2343052236 >> v1:172.21.15.94:6789/0 conn(0x7f49ac15e1e0 legacy=0x7f49ac0b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:42.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.734+0000 7f49b1c60700 1 -- 172.21.15.91:0/2343052236 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac06c3e0 con 0x7f49ac0b5e40 2023-11-24T06:08:42.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.734+0000 7f49b0c5e700 1 --2- 172.21.15.91:0/2343052236 >> v2:172.21.15.94:3300/0 conn(0x7f49ac06b400 0x7f49ac15e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:42.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.734+0000 7f49b1c60700 1 --2- 172.21.15.91:0/2343052236 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f49ac0b5e40 0x7f49ac15eb00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f49ac06ba70 tx=0x7f49a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=909bfff621153efc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:42.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.734+0000 7f49a3fff700 1 -- 172.21.15.91:0/2343052236 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a400e070 con 0x7f49ac0b5e40 2023-11-24T06:08:42.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.734+0000 7f49a3fff700 1 -- 172.21.15.91:0/2343052236 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a4003a20 con 0x7f49ac0b5e40 2023-11-24T06:08:42.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.734+0000 7f49a3fff700 1 -- 172.21.15.91:0/2343052236 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4003dd0 con 0x7f49ac0b5e40 2023-11-24T06:08:42.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.735+0000 7f49b36c3700 1 -- 172.21.15.91:0/2343052236 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f49ac0b5e40 msgr2=0x7f49ac15eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:42.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.735+0000 7f49b36c3700 1 --2- 172.21.15.91:0/2343052236 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f49ac0b5e40 0x7f49ac15eb00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f49ac06ba70 tx=0x7f49a400b330 comp rx=0 tx=0).stop 2023-11-24T06:08:42.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.735+0000 7f49b36c3700 1 -- 172.21.15.91:0/2343052236 shutdown_connections 2023-11-24T06:08:42.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.735+0000 7f49b36c3700 1 --2- 172.21.15.91:0/2343052236 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f49ac0b5e40 0x7f49ac15eb00 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.735+0000 7f49b36c3700 1 --2- 172.21.15.91:0/2343052236 >> v2:172.21.15.94:3300/0 conn(0x7f49ac06b400 0x7f49ac15e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.735+0000 7f49b36c3700 1 -- 172.21.15.91:0/2343052236 >> 172.21.15.91:0/2343052236 conn(0x7f49ac0b5070 msgr2=0x7f49ac0b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:42.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.735+0000 7f49b36c3700 1 -- 172.21.15.91:0/2343052236 shutdown_connections 2023-11-24T06:08:42.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.735+0000 7f49b36c3700 1 -- 172.21.15.91:0/2343052236 wait complete. 2023-11-24T06:08:42.742 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.736+0000 7f49b36c3700 1 Processor -- start 2023-11-24T06:08:42.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.736+0000 7f49b36c3700 1 -- start start 2023-11-24T06:08:42.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b36c3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49ac06b400 0x7f49ac14e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b36c3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f49ac0b5e40 0x7f49ac15a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b36c3700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f49ac15e1e0 0x7f49ac09d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b36c3700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac09a930 con 0x7f49ac15e1e0 2023-11-24T06:08:42.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b36c3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac14eb80 con 0x7f49ac06b400 2023-11-24T06:08:42.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b36c3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac15a9b0 con 0x7f49ac0b5e40 2023-11-24T06:08:42.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b1c60700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f49ac15e1e0 0x7f49ac09d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b1c60700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f49ac15e1e0 0x7f49ac09d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59152/0 (socket says 172.21.15.91:59152) 2023-11-24T06:08:42.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b1c60700 1 -- 172.21.15.91:0/1364316056 learned_addr learned my addr 172.21.15.91:0/1364316056 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:42.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b0c5e700 1 --2- 172.21.15.91:0/1364316056 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f49ac0b5e40 0x7f49ac15a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b145f700 1 --2- 172.21.15.91:0/1364316056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49ac06b400 0x7f49ac14e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b1c60700 1 -- 172.21.15.91:0/1364316056 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f49ac0b5e40 msgr2=0x7f49ac15a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:42.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b1c60700 1 --2- 172.21.15.91:0/1364316056 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f49ac0b5e40 0x7f49ac15a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b1c60700 1 -- 172.21.15.91:0/1364316056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49ac06b400 msgr2=0x7f49ac14e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:42.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b1c60700 1 --2- 172.21.15.91:0/1364316056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49ac06b400 0x7f49ac14e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b1c60700 1 -- 172.21.15.91:0/1364316056 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a4009b30 con 0x7f49ac15e1e0 2023-11-24T06:08:42.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b0c5e700 1 --2- 172.21.15.91:0/1364316056 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f49ac0b5e40 0x7f49ac15a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:42.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49b1c60700 1 --2- 172.21.15.91:0/1364316056 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f49ac15e1e0 0x7f49ac09d580 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f49a4015040 tx=0x7f49a4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:42.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.737+0000 7f49a1ffb700 1 -- 172.21.15.91:0/1364316056 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4004440 con 0x7f49ac15e1e0 2023-11-24T06:08:42.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.738+0000 7f49a1ffb700 1 -- 172.21.15.91:0/1364316056 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a40045a0 con 0x7f49ac15e1e0 2023-11-24T06:08:42.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.738+0000 7f49b36c3700 1 -- 172.21.15.91:0/1364316056 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ac15ac90 con 0x7f49ac15e1e0 2023-11-24T06:08:42.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.738+0000 7f49a1ffb700 1 -- 172.21.15.91:0/1364316056 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4004950 con 0x7f49ac15e1e0 2023-11-24T06:08:42.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.738+0000 7f49b36c3700 1 -- 172.21.15.91:0/1364316056 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ac09a550 con 0x7f49ac15e1e0 2023-11-24T06:08:42.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.739+0000 7f49b36c3700 1 -- 172.21.15.91:0/1364316056 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ac09aa70 con 0x7f49ac15e1e0 2023-11-24T06:08:42.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.739+0000 7f49a1ffb700 1 -- 172.21.15.91:0/1364316056 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f49a4012430 con 0x7f49ac15e1e0 2023-11-24T06:08:42.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.739+0000 7f49a1ffb700 1 --2- 172.21.15.91:0/1364316056 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f499006aed0 0x7f499006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:42.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.740+0000 7f49a1ffb700 1 -- 172.21.15.91:0/1364316056 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f49a408ab20 con 0x7f49ac15e1e0 2023-11-24T06:08:42.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.740+0000 7f49b145f700 1 --2- 172.21.15.91:0/1364316056 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f499006aed0 0x7f499006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:42.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.740+0000 7f49b145f700 1 --2- 172.21.15.91:0/1364316056 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f499006aed0 0x7f499006d390 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f499c007fd0 tx=0x7f499c007ec0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:42.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.744+0000 7f49a1ffb700 1 -- 172.21.15.91:0/1364316056 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f49ac09aa70 con 0x7f49ac15e1e0 2023-11-24T06:08:42.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.952+0000 7f49b36c3700 1 -- 172.21.15.91:0/1364316056 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f49ac09aa70 con 0x7f49ac15e1e0 2023-11-24T06:08:42.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.952+0000 7f49a1ffb700 1 -- 172.21.15.91:0/1364316056 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f49ac09aa70 con 0x7f49ac15e1e0 2023-11-24T06:08:42.958 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.954+0000 7f49b36c3700 1 -- 172.21.15.91:0/1364316056 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f499006aed0 msgr2=0x7f499006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:42.958 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.954+0000 7f49b36c3700 1 --2- 172.21.15.91:0/1364316056 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f499006aed0 0x7f499006d390 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f499c007fd0 tx=0x7f499c007ec0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.958 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.954+0000 7f49b36c3700 1 -- 172.21.15.91:0/1364316056 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f49ac15e1e0 msgr2=0x7f49ac09d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:42.958 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.954+0000 7f49b36c3700 1 --2- 172.21.15.91:0/1364316056 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f49ac15e1e0 0x7f49ac09d580 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f49a4015040 tx=0x7f49a4004ad0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.954+0000 7f49b36c3700 1 -- 172.21.15.91:0/1364316056 shutdown_connections 2023-11-24T06:08:42.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.954+0000 7f49b36c3700 1 --2- 172.21.15.91:0/1364316056 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f499006aed0 0x7f499006d390 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.954+0000 7f49b36c3700 1 --2- 172.21.15.91:0/1364316056 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f49ac06b400 0x7f49ac14e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.954+0000 7f49b36c3700 1 --2- 172.21.15.91:0/1364316056 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f49ac15e1e0 0x7f49ac09d580 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.954+0000 7f49b36c3700 1 --2- 172.21.15.91:0/1364316056 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f49ac0b5e40 0x7f49ac15a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:42.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.954+0000 7f49b36c3700 1 -- 172.21.15.91:0/1364316056 >> 172.21.15.91:0/1364316056 conn(0x7f49ac0b5070 msgr2=0x7f49ac0b5610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:42.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.954+0000 7f49b36c3700 1 -- 172.21.15.91:0/1364316056 shutdown_connections 2023-11-24T06:08:42.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:42.954+0000 7f49b36c3700 1 -- 172.21.15.91:0/1364316056 wait complete. 2023-11-24T06:08:42.959 INFO:teuthology.orchestra.run.smithi091.stderr:listed 0 entries 2023-11-24T06:08:42.970 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2023-11-24T06:08:42.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:42.970 DEBUG:teuthology.orchestra.run.smithi094:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2023-11-24T06:08:43.017 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi094.front.sepia.ceph.com 2023-11-24T06:08:43.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:08:43.017 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-11-24T06:08:43.036 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-11-24T06:08:43.037 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:08:43.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.152+0000 7f5e871d9700 1 Processor -- start 2023-11-24T06:08:43.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.152+0000 7f5e871d9700 1 -- start start 2023-11-24T06:08:43.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.153+0000 7f5e871d9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5e8006b620 0x7f5e800b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.153+0000 7f5e871d9700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5e800b5ab0 0x7f5e800b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.153+0000 7f5e871d9700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e80071f50 con 0x7f5e801478c0 2023-11-24T06:08:43.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.153+0000 7f5e871d9700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e80072230 con 0x7f5e800b5ab0 2023-11-24T06:08:43.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.153+0000 7f5e871d9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e80072510 con 0x7f5e8006b620 2023-11-24T06:08:43.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.153+0000 7f5e7ffff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5e800b5ab0 0x7f5e800b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.153+0000 7f5e84f75700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5e8006b620 0x7f5e800b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.153+0000 7f5e85776700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f5e801478c0 0x7f5e80161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.91:55886/0 (socket says 172.21.15.91:55886) 2023-11-24T06:08:43.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.153+0000 7f5e85776700 1 -- 172.21.15.91:0/2967174623 learned_addr learned my addr 172.21.15.91:0/2967174623 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:43.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.153+0000 7f5e7ffff700 1 -- 172.21.15.91:0/2967174623 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5e8006b620 msgr2=0x7f5e800b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:43.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.153+0000 7f5e7ffff700 1 --2- 172.21.15.91:0/2967174623 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5e8006b620 0x7f5e800b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.153+0000 7f5e7ffff700 1 -- 172.21.15.91:0/2967174623 >> v1:172.21.15.94:6789/0 conn(0x7f5e801478c0 legacy=0x7f5e80161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:43.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.154+0000 7f5e7ffff700 1 -- 172.21.15.91:0/2967174623 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e800727f0 con 0x7f5e800b5ab0 2023-11-24T06:08:43.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.154+0000 7f5e84f75700 1 --2- 172.21.15.91:0/2967174623 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5e8006b620 0x7f5e800b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:08:43.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.154+0000 7f5e7ffff700 1 --2- 172.21.15.91:0/2967174623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e800b5ab0 0x7f5e800b3f80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009840 tx=0x7f5e6c009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddb35eb45f533193 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:43.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.154+0000 7f5e7f7fe700 1 -- 172.21.15.91:0/2967174623 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c011830 con 0x7f5e800b5ab0 2023-11-24T06:08:43.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.154+0000 7f5e7f7fe700 1 -- 172.21.15.91:0/2967174623 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6c011990 con 0x7f5e800b5ab0 2023-11-24T06:08:43.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.154+0000 7f5e7f7fe700 1 -- 172.21.15.91:0/2967174623 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c011d20 con 0x7f5e800b5ab0 2023-11-24T06:08:43.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.154+0000 7f5e871d9700 1 -- 172.21.15.91:0/2967174623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e800b5ab0 msgr2=0x7f5e800b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:43.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.154+0000 7f5e871d9700 1 --2- 172.21.15.91:0/2967174623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e800b5ab0 0x7f5e800b3f80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009840 tx=0x7f5e6c009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:43.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.154+0000 7f5e871d9700 1 -- 172.21.15.91:0/2967174623 shutdown_connections 2023-11-24T06:08:43.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.154+0000 7f5e871d9700 1 --2- 172.21.15.91:0/2967174623 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5e8006b620 0x7f5e800b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.154+0000 7f5e871d9700 1 --2- 172.21.15.91:0/2967174623 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e800b5ab0 0x7f5e800b3f80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.154+0000 7f5e871d9700 1 -- 172.21.15.91:0/2967174623 >> 172.21.15.91:0/2967174623 conn(0x7f5e800b2740 msgr2=0x7f5e800b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:43.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.154+0000 7f5e871d9700 1 -- 172.21.15.91:0/2967174623 shutdown_connections 2023-11-24T06:08:43.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.154+0000 7f5e871d9700 1 -- 172.21.15.91:0/2967174623 wait complete. 2023-11-24T06:08:43.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.156+0000 7f5e871d9700 1 Processor -- start 2023-11-24T06:08:43.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.156+0000 7f5e871d9700 1 -- start start 2023-11-24T06:08:43.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.157+0000 7f5e871d9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e8006b620 0x7f5e801e74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.157+0000 7f5e871d9700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5e800b5ab0 0x7f5e800cf860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.157+0000 7f5e871d9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5e801478c0 0x7f5e801e8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.157+0000 7f5e871d9700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e801e7e10 con 0x7f5e800b5ab0 2023-11-24T06:08:43.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.157+0000 7f5e871d9700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e801e80f0 con 0x7f5e8006b620 2023-11-24T06:08:43.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.157+0000 7f5e871d9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e801e83d0 con 0x7f5e801478c0 2023-11-24T06:08:43.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.157+0000 7f5e7ffff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5e800b5ab0 0x7f5e800cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.157+0000 7f5e7ffff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5e800b5ab0 0x7f5e800cf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59166/0 (socket says 172.21.15.91:59166) 2023-11-24T06:08:43.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.157+0000 7f5e7ffff700 1 -- 172.21.15.91:0/998124691 learned_addr learned my addr 172.21.15.91:0/998124691 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:43.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.157+0000 7f5e84f75700 1 --2- 172.21.15.91:0/998124691 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e8006b620 0x7f5e801e74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.157+0000 7f5e85776700 1 --2- 172.21.15.91:0/998124691 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5e801478c0 0x7f5e801e8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.158+0000 7f5e7ffff700 1 -- 172.21.15.91:0/998124691 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5e801478c0 msgr2=0x7f5e801e8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:43.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.158+0000 7f5e7ffff700 1 --2- 172.21.15.91:0/998124691 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5e801478c0 0x7f5e801e8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.158+0000 7f5e7ffff700 1 -- 172.21.15.91:0/998124691 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e8006b620 msgr2=0x7f5e801e74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:43.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.158+0000 7f5e7ffff700 1 --2- 172.21.15.91:0/998124691 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e8006b620 0x7f5e801e74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.158+0000 7f5e7ffff700 1 -- 172.21.15.91:0/998124691 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e740094a0 con 0x7f5e800b5ab0 2023-11-24T06:08:43.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.158+0000 7f5e7ffff700 1 --2- 172.21.15.91:0/998124691 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5e800b5ab0 0x7f5e800cf860 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c000c00 tx=0x7f5e6c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:43.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.158+0000 7f5e7d7fa700 1 -- 172.21.15.91:0/998124691 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c0036a0 con 0x7f5e800b5ab0 2023-11-24T06:08:43.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.158+0000 7f5e7d7fa700 1 -- 172.21.15.91:0/998124691 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6c01f070 con 0x7f5e800b5ab0 2023-11-24T06:08:43.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.158+0000 7f5e871d9700 1 -- 172.21.15.91:0/998124691 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6c0094a0 con 0x7f5e800b5ab0 2023-11-24T06:08:43.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.158+0000 7f5e7d7fa700 1 -- 172.21.15.91:0/998124691 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c003a40 con 0x7f5e800b5ab0 2023-11-24T06:08:43.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.158+0000 7f5e871d9700 1 -- 172.21.15.91:0/998124691 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e801e8990 con 0x7f5e800b5ab0 2023-11-24T06:08:43.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.159+0000 7f5e871d9700 1 -- 172.21.15.91:0/998124691 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e801e8b20 con 0x7f5e800b5ab0 2023-11-24T06:08:43.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.159+0000 7f5e7d7fa700 1 -- 172.21.15.91:0/998124691 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5e6c003ba0 con 0x7f5e800b5ab0 2023-11-24T06:08:43.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.160+0000 7f5e7d7fa700 1 --2- 172.21.15.91:0/998124691 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f5e6406aed0 0x7f5e6406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.160+0000 7f5e84f75700 1 --2- 172.21.15.91:0/998124691 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f5e6406aed0 0x7f5e6406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.160+0000 7f5e7d7fa700 1 -- 172.21.15.91:0/998124691 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5e6c021070 con 0x7f5e800b5ab0 2023-11-24T06:08:43.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.160+0000 7f5e84f75700 1 --2- 172.21.15.91:0/998124691 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f5e6406aed0 0x7f5e6406d390 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5e801e3500 tx=0x7f5e74009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:43.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.164+0000 7f5e7d7fa700 1 -- 172.21.15.91:0/998124691 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e6c090020 con 0x7f5e800b5ab0 2023-11-24T06:08:43.402 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.398+0000 7f5e871d9700 1 -- 172.21.15.91:0/998124691 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e80068670 con 0x7f5e800b5ab0 2023-11-24T06:08:43.402 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.398+0000 7f5e7d7fa700 1 -- 172.21.15.91:0/998124691 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4907 (secure 0 0 0) 0x7f5e6c015020 con 0x7f5e800b5ab0 2023-11-24T06:08:43.403 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:43.403 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1329219967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1329219967},{"type":"v1","addr":"172.21.15.94:6834","nonce":1329219967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3439524864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3439524864},{"type":"v1","addr":"172.21.15.94:6835","nonce":3439524864}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6836/3390368219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6834","nonce":3390368219},{"type":"v1","addr":"172.21.15.91:6836","nonce":3390368219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:08:00.798681+0000","modified":"2023-11-24T06:08:04.132764+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.91:6837/1644079645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":1644079645},{"type":"v1","addr":"172.21.15.91:6837","nonce":1644079645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T06:08:43.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.400+0000 7f5e871d9700 1 -- 172.21.15.91:0/998124691 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f5e6406aed0 msgr2=0x7f5e6406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:43.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.400+0000 7f5e871d9700 1 --2- 172.21.15.91:0/998124691 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f5e6406aed0 0x7f5e6406d390 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5e801e3500 tx=0x7f5e74009210 comp rx=0 tx=0).stop 2023-11-24T06:08:43.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.400+0000 7f5e871d9700 1 -- 172.21.15.91:0/998124691 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5e800b5ab0 msgr2=0x7f5e800cf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:43.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.400+0000 7f5e871d9700 1 --2- 172.21.15.91:0/998124691 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5e800b5ab0 0x7f5e800cf860 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c000c00 tx=0x7f5e6c004750 comp rx=0 tx=0).stop 2023-11-24T06:08:43.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.400+0000 7f5e871d9700 1 -- 172.21.15.91:0/998124691 shutdown_connections 2023-11-24T06:08:43.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.400+0000 7f5e871d9700 1 --2- 172.21.15.91:0/998124691 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f5e6406aed0 0x7f5e6406d390 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.401+0000 7f5e871d9700 1 --2- 172.21.15.91:0/998124691 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5e8006b620 0x7f5e801e74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.401+0000 7f5e871d9700 1 --2- 172.21.15.91:0/998124691 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5e801478c0 0x7f5e801e8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.401+0000 7f5e871d9700 1 --2- 172.21.15.91:0/998124691 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5e800b5ab0 0x7f5e800cf860 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.401+0000 7f5e871d9700 1 -- 172.21.15.91:0/998124691 >> 172.21.15.91:0/998124691 conn(0x7f5e800b2740 msgr2=0x7f5e801e5980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:43.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.401+0000 7f5e871d9700 1 -- 172.21.15.91:0/998124691 shutdown_connections 2023-11-24T06:08:43.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.401+0000 7f5e871d9700 1 -- 172.21.15.91:0/998124691 wait complete. 2023-11-24T06:08:43.406 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 8 2023-11-24T06:08:43.417 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:43.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.533+0000 7f6eab1eb700 1 Processor -- start 2023-11-24T06:08:43.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.533+0000 7f6eab1eb700 1 -- start start 2023-11-24T06:08:43.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.534+0000 7f6eab1eb700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f6ea406b620 0x7f6ea40b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.534+0000 7f6eab1eb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6ea40b5ab0 0x7f6ea40b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.534+0000 7f6eab1eb700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6ea41478c0 0x7f6ea4161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.534+0000 7f6eab1eb700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea4071f50 con 0x7f6ea406b620 2023-11-24T06:08:43.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.534+0000 7f6eab1eb700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea4072230 con 0x7f6ea41478c0 2023-11-24T06:08:43.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.534+0000 7f6eab1eb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea4072510 con 0x7f6ea40b5ab0 2023-11-24T06:08:43.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.534+0000 7f6ea8f87700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f6ea406b620 0x7f6ea40b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.534+0000 7f6ea8f87700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f6ea406b620 0x7f6ea40b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59170/0 (socket says 172.21.15.91:59170) 2023-11-24T06:08:43.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.534+0000 7f6ea8f87700 1 -- 172.21.15.91:0/3254648419 learned_addr learned my addr 172.21.15.91:0/3254648419 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:43.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.534+0000 7f6ea9788700 1 --2- 172.21.15.91:0/3254648419 >> v2:172.21.15.94:3300/0 conn(0x7f6ea41478c0 0x7f6ea4161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.534+0000 7f6ea3fff700 1 --2- 172.21.15.91:0/3254648419 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6ea40b5ab0 0x7f6ea40b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6ea8f87700 1 -- 172.21.15.91:0/3254648419 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6ea40b5ab0 msgr2=0x7f6ea40b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:43.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6ea8f87700 1 --2- 172.21.15.91:0/3254648419 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6ea40b5ab0 0x7f6ea40b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6ea8f87700 1 -- 172.21.15.91:0/3254648419 >> v2:172.21.15.94:3300/0 conn(0x7f6ea41478c0 msgr2=0x7f6ea4161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:43.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6ea8f87700 1 --2- 172.21.15.91:0/3254648419 >> v2:172.21.15.94:3300/0 conn(0x7f6ea41478c0 0x7f6ea4161ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6ea8f87700 1 -- 172.21.15.91:0/3254648419 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea40727f0 con 0x7f6ea406b620 2023-11-24T06:08:43.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6ea3fff700 1 --2- 172.21.15.91:0/3254648419 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6ea40b5ab0 0x7f6ea40b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:43.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6ea8f87700 1 --2- 172.21.15.91:0/3254648419 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6ea406b620 0x7f6ea40b3a40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40747c0 tx=0x7f6e98009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48452b4cc927fa57 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:43.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6ea37fe700 1 -- 172.21.15.91:0/3254648419 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98011830 con 0x7f6ea406b620 2023-11-24T06:08:43.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6ea37fe700 1 -- 172.21.15.91:0/3254648419 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e98011990 con 0x7f6ea406b620 2023-11-24T06:08:43.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6ea37fe700 1 -- 172.21.15.91:0/3254648419 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98011d20 con 0x7f6ea406b620 2023-11-24T06:08:43.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6eab1eb700 1 -- 172.21.15.91:0/3254648419 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6ea406b620 msgr2=0x7f6ea40b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:43.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6eab1eb700 1 --2- 172.21.15.91:0/3254648419 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6ea406b620 0x7f6ea40b3a40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6ea40747c0 tx=0x7f6e98009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6eab1eb700 1 -- 172.21.15.91:0/3254648419 shutdown_connections 2023-11-24T06:08:43.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6eab1eb700 1 --2- 172.21.15.91:0/3254648419 >> v2:172.21.15.94:3300/0 conn(0x7f6ea41478c0 0x7f6ea4161ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6eab1eb700 1 --2- 172.21.15.91:0/3254648419 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6ea406b620 0x7f6ea40b3a40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6eab1eb700 1 --2- 172.21.15.91:0/3254648419 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6ea40b5ab0 0x7f6ea40b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6eab1eb700 1 -- 172.21.15.91:0/3254648419 >> 172.21.15.91:0/3254648419 conn(0x7f6ea40b2740 msgr2=0x7f6ea40b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:43.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6eab1eb700 1 -- 172.21.15.91:0/3254648419 shutdown_connections 2023-11-24T06:08:43.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.535+0000 7f6eab1eb700 1 -- 172.21.15.91:0/3254648419 wait complete. 2023-11-24T06:08:43.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.537+0000 7f6eab1eb700 1 Processor -- start 2023-11-24T06:08:43.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.537+0000 7f6eab1eb700 1 -- start start 2023-11-24T06:08:43.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.537+0000 7f6eab1eb700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6ea406b620 0x7f6ea40ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.537+0000 7f6eab1eb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6ea40b5ab0 0x7f6ea40c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.537+0000 7f6eab1eb700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ea41478c0 0x7f6ea40b0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6eab1eb700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea40ca6e0 con 0x7f6ea406b620 2023-11-24T06:08:43.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6eab1eb700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea41e4d70 con 0x7f6ea41478c0 2023-11-24T06:08:43.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6eab1eb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea41f7eb0 con 0x7f6ea40b5ab0 2023-11-24T06:08:43.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6ea8f87700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6ea406b620 0x7f6ea40ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6ea8f87700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6ea406b620 0x7f6ea40ca1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59176/0 (socket says 172.21.15.91:59176) 2023-11-24T06:08:43.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6ea8f87700 1 -- 172.21.15.91:0/1533297487 learned_addr learned my addr 172.21.15.91:0/1533297487 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:43.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6ea3fff700 1 --2- 172.21.15.91:0/1533297487 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6ea40b5ab0 0x7f6ea40c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6ea8f87700 1 -- 172.21.15.91:0/1533297487 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6ea40b5ab0 msgr2=0x7f6ea40c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:43.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6ea8f87700 1 --2- 172.21.15.91:0/1533297487 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6ea40b5ab0 0x7f6ea40c70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6ea8f87700 1 -- 172.21.15.91:0/1533297487 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ea41478c0 msgr2=0x7f6ea40b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:43.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6ea8f87700 1 --2- 172.21.15.91:0/1533297487 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ea41478c0 0x7f6ea40b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6ea8f87700 1 -- 172.21.15.91:0/1533297487 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e98009550 con 0x7f6ea406b620 2023-11-24T06:08:43.548 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6ea8f87700 1 --2- 172.21.15.91:0/1533297487 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6ea406b620 0x7f6ea40ca1a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009c10 tx=0x7f6e98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:43.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6ea17fa700 1 -- 172.21.15.91:0/1533297487 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98004300 con 0x7f6ea406b620 2023-11-24T06:08:43.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6ea17fa700 1 -- 172.21.15.91:0/1533297487 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9801f070 con 0x7f6ea406b620 2023-11-24T06:08:43.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6ea17fa700 1 -- 172.21.15.91:0/1533297487 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e980036a0 con 0x7f6ea406b620 2023-11-24T06:08:43.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6eab1eb700 1 -- 172.21.15.91:0/1533297487 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea40c75f0 con 0x7f6ea406b620 2023-11-24T06:08:43.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.538+0000 7f6eab1eb700 1 -- 172.21.15.91:0/1533297487 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea41f7580 con 0x7f6ea406b620 2023-11-24T06:08:43.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.539+0000 7f6eab1eb700 1 -- 172.21.15.91:0/1533297487 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea4068670 con 0x7f6ea406b620 2023-11-24T06:08:43.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.540+0000 7f6ea17fa700 1 -- 172.21.15.91:0/1533297487 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6e98021030 con 0x7f6ea406b620 2023-11-24T06:08:43.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.540+0000 7f6ea17fa700 1 --2- 172.21.15.91:0/1533297487 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6e8806aed0 0x7f6e8806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.549 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.540+0000 7f6ea17fa700 1 -- 172.21.15.91:0/1533297487 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6e9808a890 con 0x7f6ea406b620 2023-11-24T06:08:43.550 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.540+0000 7f6ea3fff700 1 --2- 172.21.15.91:0/1533297487 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6e8806aed0 0x7f6e8806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.550 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.540+0000 7f6ea3fff700 1 --2- 172.21.15.91:0/1533297487 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6e8806aed0 0x7f6e8806d390 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6ea41f6ef0 tx=0x7f6e90008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:43.550 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.544+0000 7f6ea17fa700 1 -- 172.21.15.91:0/1533297487 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e98055950 con 0x7f6ea406b620 2023-11-24T06:08:43.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.735+0000 7f6eab1eb700 1 -- 172.21.15.91:0/1533297487 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ea406bd40 con 0x7f6ea406b620 2023-11-24T06:08:43.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.736+0000 7f6ea17fa700 1 -- 172.21.15.91:0/1533297487 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16671 (secure 0 0 0) 0x7f6e98015020 con 0x7f6ea406b620 2023-11-24T06:08:43.740 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:43.741 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":22,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:42.798927+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:43.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.738+0000 7f6eab1eb700 1 -- 172.21.15.91:0/1533297487 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6e8806aed0 msgr2=0x7f6e8806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:43.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.738+0000 7f6eab1eb700 1 --2- 172.21.15.91:0/1533297487 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6e8806aed0 0x7f6e8806d390 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6ea41f6ef0 tx=0x7f6e90008040 comp rx=0 tx=0).stop 2023-11-24T06:08:43.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.739+0000 7f6eab1eb700 1 -- 172.21.15.91:0/1533297487 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6ea406b620 msgr2=0x7f6ea40ca1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:43.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.739+0000 7f6eab1eb700 1 --2- 172.21.15.91:0/1533297487 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6ea406b620 0x7f6ea40ca1a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009c10 tx=0x7f6e98004750 comp rx=0 tx=0).stop 2023-11-24T06:08:43.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.739+0000 7f6eab1eb700 1 -- 172.21.15.91:0/1533297487 shutdown_connections 2023-11-24T06:08:43.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.739+0000 7f6eab1eb700 1 --2- 172.21.15.91:0/1533297487 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6e8806aed0 0x7f6e8806d390 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.739+0000 7f6eab1eb700 1 --2- 172.21.15.91:0/1533297487 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6ea41478c0 0x7f6ea40b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.739+0000 7f6eab1eb700 1 --2- 172.21.15.91:0/1533297487 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6ea406b620 0x7f6ea40ca1a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.739+0000 7f6eab1eb700 1 --2- 172.21.15.91:0/1533297487 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6ea40b5ab0 0x7f6ea40c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.739+0000 7f6eab1eb700 1 -- 172.21.15.91:0/1533297487 >> 172.21.15.91:0/1533297487 conn(0x7f6ea40b2740 msgr2=0x7f6ea40b14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:43.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.739+0000 7f6eab1eb700 1 -- 172.21.15.91:0/1533297487 shutdown_connections 2023-11-24T06:08:43.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.739+0000 7f6eab1eb700 1 -- 172.21.15.91:0/1533297487 wait complete. 2023-11-24T06:08:43.755 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:07:58.710106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:07:59.774819+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:08:43.756 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:08:43.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.877+0000 7f02b96d3700 1 Processor -- start 2023-11-24T06:08:43.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.877+0000 7f02b96d3700 1 -- start start 2023-11-24T06:08:43.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.878+0000 7f02b96d3700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f02b415e0e0 0x7f02b40d2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.878+0000 7f02b96d3700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f02b40d2960 0x7f02b40d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.878+0000 7f02b96d3700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b40d1010 con 0x7f02b40d0bc0 2023-11-24T06:08:43.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.878+0000 7f02b96d3700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b40cf800 con 0x7f02b415e0e0 2023-11-24T06:08:43.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.878+0000 7f02b96d3700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b40d4ac0 con 0x7f02b40d2960 2023-11-24T06:08:43.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.878+0000 7f02b2ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f02b415e0e0 0x7f02b40d2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.878+0000 7f02b37fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f02b40d0bc0 0x7f02b40b8f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54122/0 (socket says 172.21.15.91:54122) 2023-11-24T06:08:43.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.878+0000 7f02b37fe700 1 -- 172.21.15.91:0/1206089526 learned_addr learned my addr 172.21.15.91:0/1206089526 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:43.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.878+0000 7f02b2ffd700 1 -- 172.21.15.91:0/1206089526 >> v2:172.21.15.94:3300/0 conn(0x7f02b40d2960 msgr2=0x7f02b40d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:43.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.878+0000 7f02b27fc700 1 --2- 172.21.15.91:0/1206089526 >> v2:172.21.15.94:3300/0 conn(0x7f02b40d2960 0x7f02b40d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.878+0000 7f02b2ffd700 1 --2- 172.21.15.91:0/1206089526 >> v2:172.21.15.94:3300/0 conn(0x7f02b40d2960 0x7f02b40d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.878+0000 7f02b1ffb700 1 -- 172.21.15.91:0/1206089526 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857797744 0 0) 0x7f02ac001490 con 0x7f02b40d0bc0 2023-11-24T06:08:43.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.878+0000 7f02b2ffd700 1 -- 172.21.15.91:0/1206089526 >> v1:172.21.15.91:6789/0 conn(0x7f02b40d0bc0 legacy=0x7f02b40b8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:43.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.878+0000 7f02b2ffd700 1 -- 172.21.15.91:0/1206089526 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b415e4c0 con 0x7f02b415e0e0 2023-11-24T06:08:43.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.878+0000 7f02b27fc700 1 --2- 172.21.15.91:0/1206089526 >> v2:172.21.15.94:3300/0 conn(0x7f02b40d2960 0x7f02b40d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:43.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.878+0000 7f02b2ffd700 1 --2- 172.21.15.91:0/1206089526 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f02b415e0e0 0x7f02b40d2420 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f02b408c630 tx=0x7f02a8009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0df76743a390045 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:43.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.879+0000 7f02b1ffb700 1 -- 172.21.15.91:0/1206089526 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8011830 con 0x7f02b415e0e0 2023-11-24T06:08:43.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.879+0000 7f02b1ffb700 1 -- 172.21.15.91:0/1206089526 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a8011990 con 0x7f02b415e0e0 2023-11-24T06:08:43.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.879+0000 7f02b1ffb700 1 -- 172.21.15.91:0/1206089526 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8011d20 con 0x7f02b415e0e0 2023-11-24T06:08:43.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.879+0000 7f02b96d3700 1 -- 172.21.15.91:0/1206089526 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f02b415e0e0 msgr2=0x7f02b40d2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:43.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.879+0000 7f02b96d3700 1 --2- 172.21.15.91:0/1206089526 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f02b415e0e0 0x7f02b40d2420 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f02b408c630 tx=0x7f02a8009b20 comp rx=0 tx=0).stop 2023-11-24T06:08:43.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.879+0000 7f02b96d3700 1 -- 172.21.15.91:0/1206089526 shutdown_connections 2023-11-24T06:08:43.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.879+0000 7f02b96d3700 1 --2- 172.21.15.91:0/1206089526 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f02b415e0e0 0x7f02b40d2420 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.879+0000 7f02b96d3700 1 --2- 172.21.15.91:0/1206089526 >> v2:172.21.15.94:3300/0 conn(0x7f02b40d2960 0x7f02b40d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.879+0000 7f02b96d3700 1 -- 172.21.15.91:0/1206089526 >> 172.21.15.91:0/1206089526 conn(0x7f02b409d110 msgr2=0x7f02b40d3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:43.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.879+0000 7f02b96d3700 1 -- 172.21.15.91:0/1206089526 shutdown_connections 2023-11-24T06:08:43.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.879+0000 7f02b96d3700 1 -- 172.21.15.91:0/1206089526 wait complete. 2023-11-24T06:08:43.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.880+0000 7f02b96d3700 1 Processor -- start 2023-11-24T06:08:43.888 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.880+0000 7f02b96d3700 1 -- start start 2023-11-24T06:08:43.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.881+0000 7f02b96d3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02b40d0bc0 0x7f02b41e8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.881+0000 7f02b96d3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f02b40d2960 0x7f02b415d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.881+0000 7f02b96d3700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f02b415e0e0 0x7f02b41e6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.881+0000 7f02b96d3700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b41e7d40 con 0x7f02b415e0e0 2023-11-24T06:08:43.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.881+0000 7f02b96d3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b41e70c0 con 0x7f02b40d0bc0 2023-11-24T06:08:43.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.881+0000 7f02b96d3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b41e73a0 con 0x7f02b40d2960 2023-11-24T06:08:43.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.881+0000 7f02b37fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f02b415e0e0 0x7f02b41e6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.881+0000 7f02b37fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f02b415e0e0 0x7f02b41e6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59194/0 (socket says 172.21.15.91:59194) 2023-11-24T06:08:43.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.881+0000 7f02b37fe700 1 -- 172.21.15.91:0/1336741876 learned_addr learned my addr 172.21.15.91:0/1336741876 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:43.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.881+0000 7f02b2ffd700 1 --2- 172.21.15.91:0/1336741876 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02b40d0bc0 0x7f02b41e8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.881+0000 7f02b27fc700 1 --2- 172.21.15.91:0/1336741876 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f02b40d2960 0x7f02b415d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.882+0000 7f02b37fe700 1 -- 172.21.15.91:0/1336741876 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f02b40d2960 msgr2=0x7f02b415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:43.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.882+0000 7f02b37fe700 1 --2- 172.21.15.91:0/1336741876 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f02b40d2960 0x7f02b415d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.882+0000 7f02b37fe700 1 -- 172.21.15.91:0/1336741876 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02b40d0bc0 msgr2=0x7f02b41e8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:43.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.882+0000 7f02b37fe700 1 --2- 172.21.15.91:0/1336741876 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02b40d0bc0 0x7f02b41e8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:43.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.882+0000 7f02b37fe700 1 -- 172.21.15.91:0/1336741876 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a80094a0 con 0x7f02b415e0e0 2023-11-24T06:08:43.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.882+0000 7f02b37fe700 1 --2- 172.21.15.91:0/1336741876 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f02b415e0e0 0x7f02b41e6600 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00cb60 tx=0x7f02ac00ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:43.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.882+0000 7f029b7fe700 1 -- 172.21.15.91:0/1336741876 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ac00a9e0 con 0x7f02b415e0e0 2023-11-24T06:08:43.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.882+0000 7f029b7fe700 1 -- 172.21.15.91:0/1336741876 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02ac00ab40 con 0x7f02b415e0e0 2023-11-24T06:08:43.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.882+0000 7f029b7fe700 1 -- 172.21.15.91:0/1336741876 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ac0070d0 con 0x7f02b415e0e0 2023-11-24T06:08:43.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.882+0000 7f02b96d3700 1 -- 172.21.15.91:0/1336741876 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b41e7680 con 0x7f02b415e0e0 2023-11-24T06:08:43.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.882+0000 7f02b96d3700 1 -- 172.21.15.91:0/1336741876 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b41e86a0 con 0x7f02b415e0e0 2023-11-24T06:08:43.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.883+0000 7f02b96d3700 1 -- 172.21.15.91:0/1336741876 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b41e7fb0 con 0x7f02b415e0e0 2023-11-24T06:08:43.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.883+0000 7f029b7fe700 1 -- 172.21.15.91:0/1336741876 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f02ac007230 con 0x7f02b415e0e0 2023-11-24T06:08:43.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.884+0000 7f029b7fe700 1 --2- 172.21.15.91:0/1336741876 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f02a006af00 0x7f02a006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:43.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.884+0000 7f029b7fe700 1 -- 172.21.15.91:0/1336741876 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f02ac088cf0 con 0x7f02b415e0e0 2023-11-24T06:08:43.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.884+0000 7f02b2ffd700 1 --2- 172.21.15.91:0/1336741876 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f02a006af00 0x7f02a006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:43.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.884+0000 7f02b2ffd700 1 --2- 172.21.15.91:0/1336741876 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f02a006af00 0x7f02a006d3c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f02a80090b0 tx=0x7f02a8008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:43.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:43.888+0000 7f029b7fe700 1 -- 172.21.15.91:0/1336741876 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f02ac08d070 con 0x7f02b415e0e0 2023-11-24T06:08:44.081 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.077+0000 7f02b96d3700 1 -- 172.21.15.91:0/1336741876 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f02b40d1010 con 0x7f02b415e0e0 2023-11-24T06:08:44.082 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.078+0000 7f029b7fe700 1 -- 172.21.15.91:0/1336741876 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f02b40d1010 con 0x7f02b415e0e0 2023-11-24T06:08:44.082 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:08:44.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.080+0000 7f02b96d3700 1 -- 172.21.15.91:0/1336741876 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f02a006af00 msgr2=0x7f02a006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.080+0000 7f02b96d3700 1 --2- 172.21.15.91:0/1336741876 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f02a006af00 0x7f02a006d3c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f02a80090b0 tx=0x7f02a8008030 comp rx=0 tx=0).stop 2023-11-24T06:08:44.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.080+0000 7f02b96d3700 1 -- 172.21.15.91:0/1336741876 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f02b415e0e0 msgr2=0x7f02b41e6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.080+0000 7f02b96d3700 1 --2- 172.21.15.91:0/1336741876 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f02b415e0e0 0x7f02b41e6600 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00cb60 tx=0x7f02ac00ce70 comp rx=0 tx=0).stop 2023-11-24T06:08:44.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.080+0000 7f02b96d3700 1 -- 172.21.15.91:0/1336741876 shutdown_connections 2023-11-24T06:08:44.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.080+0000 7f02b96d3700 1 --2- 172.21.15.91:0/1336741876 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f02a006af00 0x7f02a006d3c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.080+0000 7f02b96d3700 1 --2- 172.21.15.91:0/1336741876 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f02b40d0bc0 0x7f02b41e8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.080+0000 7f02b96d3700 1 --2- 172.21.15.91:0/1336741876 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f02b415e0e0 0x7f02b41e6600 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.080+0000 7f02b96d3700 1 --2- 172.21.15.91:0/1336741876 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f02b40d2960 0x7f02b415d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.080+0000 7f02b96d3700 1 -- 172.21.15.91:0/1336741876 >> 172.21.15.91:0/1336741876 conn(0x7f02b409d110 msgr2=0x7f02b415f190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:44.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.080+0000 7f02b96d3700 1 -- 172.21.15.91:0/1336741876 shutdown_connections 2023-11-24T06:08:44.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.080+0000 7f02b96d3700 1 -- 172.21.15.91:0/1336741876 wait complete. 2023-11-24T06:08:44.096 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:08:44.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.215+0000 7f01acac9700 1 Processor -- start 2023-11-24T06:08:44.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.215+0000 7f01acac9700 1 -- start start 2023-11-24T06:08:44.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.216+0000 7f01acac9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01a80d0870 0x7f01a80d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.216+0000 7f01acac9700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f01a80d49f0 0x7f01a80b6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.216+0000 7f01acac9700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80b5370 con 0x7f01a80d0c50 2023-11-24T06:08:44.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.216+0000 7f01acac9700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a80d2f80 con 0x7f01a80d49f0 2023-11-24T06:08:44.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.216+0000 7f01acac9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80cff70 con 0x7f01a80d0870 2023-11-24T06:08:44.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.216+0000 7f01a5d9b700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f01a80d0c50 0x7f01a80cf860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54132/0 (socket says 172.21.15.91:54132) 2023-11-24T06:08:44.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.216+0000 7f01a5d9b700 1 -- 172.21.15.91:0/3916785833 learned_addr learned my addr 172.21.15.91:0/3916785833 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:44.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.216+0000 7f01a6d9d700 1 --2- 172.21.15.91:0/3916785833 >> v2:172.21.15.94:3300/0 conn(0x7f01a80d49f0 0x7f01a80b6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.216+0000 7f01a659c700 1 --2- 172.21.15.91:0/3916785833 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01a80d0870 0x7f01a80d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.216+0000 7f01a559a700 1 -- 172.21.15.91:0/3916785833 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259081891 0 0) 0x7f01a80b5370 con 0x7f01a80d0c50 2023-11-24T06:08:44.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.216+0000 7f01a559a700 1 -- 172.21.15.91:0/3916785833 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01900035b0 con 0x7f01a80d0c50 2023-11-24T06:08:44.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.216+0000 7f01a659c700 1 -- 172.21.15.91:0/3916785833 >> v2:172.21.15.94:3300/0 conn(0x7f01a80d49f0 msgr2=0x7f01a80b6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:44.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.216+0000 7f01a659c700 1 --2- 172.21.15.91:0/3916785833 >> v2:172.21.15.94:3300/0 conn(0x7f01a80d49f0 0x7f01a80b6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.216+0000 7f01a659c700 1 -- 172.21.15.91:0/3916785833 >> v1:172.21.15.91:6789/0 conn(0x7f01a80d0c50 legacy=0x7f01a80cf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.216+0000 7f01a559a700 1 -- 172.21.15.91:0/3916785833 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2663807215 0 0) 0x7f01900035b0 con 0x7f01a80d0c50 2023-11-24T06:08:44.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.216+0000 7f01a659c700 1 -- 172.21.15.91:0/3916785833 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a80ceae0 con 0x7f01a80d0870 2023-11-24T06:08:44.223 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.217+0000 7f01a659c700 1 --2- 172.21.15.91:0/3916785833 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01a80d0870 0x7f01a80d26c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f019c009840 tx=0x7f019c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c05518e4a4597553 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:44.223 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.217+0000 7f01a559a700 1 -- 172.21.15.91:0/3916785833 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0198001670 con 0x7f01a80d0870 2023-11-24T06:08:44.223 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.217+0000 7f01a559a700 1 -- 172.21.15.91:0/3916785833 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c004030 con 0x7f01a80d0870 2023-11-24T06:08:44.223 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.217+0000 7f01a559a700 1 -- 172.21.15.91:0/3916785833 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c0043c0 con 0x7f01a80d0870 2023-11-24T06:08:44.223 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.217+0000 7f01acac9700 1 -- 172.21.15.91:0/3916785833 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01a80d0870 msgr2=0x7f01a80d26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.223 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.217+0000 7f01acac9700 1 --2- 172.21.15.91:0/3916785833 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01a80d0870 0x7f01a80d26c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f019c009840 tx=0x7f019c009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:44.223 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.217+0000 7f01acac9700 1 -- 172.21.15.91:0/3916785833 shutdown_connections 2023-11-24T06:08:44.223 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.217+0000 7f01acac9700 1 --2- 172.21.15.91:0/3916785833 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01a80d0870 0x7f01a80d26c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.223 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.217+0000 7f01acac9700 1 --2- 172.21.15.91:0/3916785833 >> v2:172.21.15.94:3300/0 conn(0x7f01a80d49f0 0x7f01a80b6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.223 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.217+0000 7f01acac9700 1 -- 172.21.15.91:0/3916785833 >> 172.21.15.91:0/3916785833 conn(0x7f01a80d3ed0 msgr2=0x7f01a80d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:44.224 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.217+0000 7f01acac9700 1 -- 172.21.15.91:0/3916785833 shutdown_connections 2023-11-24T06:08:44.224 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.217+0000 7f01acac9700 1 -- 172.21.15.91:0/3916785833 wait complete. 2023-11-24T06:08:44.226 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.219+0000 7f01acac9700 1 Processor -- start 2023-11-24T06:08:44.226 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.219+0000 7f01acac9700 1 -- start start 2023-11-24T06:08:44.226 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.219+0000 7f01acac9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01a80d0870 0x7f01a815a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.219+0000 7f01acac9700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f01a80d0c50 0x7f01a80b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.219+0000 7f01acac9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01a80d49f0 0x7f01a80b4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.219+0000 7f01acac9700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80b3490 con 0x7f01a80d0c50 2023-11-24T06:08:44.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.219+0000 7f01acac9700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80b3770 con 0x7f01a80d0870 2023-11-24T06:08:44.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.219+0000 7f01acac9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80b3a50 con 0x7f01a80d49f0 2023-11-24T06:08:44.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.220+0000 7f01a5d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f01a80d0c50 0x7f01a80b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.220+0000 7f01a5d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f01a80d0c50 0x7f01a80b13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59206/0 (socket says 172.21.15.91:59206) 2023-11-24T06:08:44.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.220+0000 7f01a5d9b700 1 -- 172.21.15.91:0/527513774 learned_addr learned my addr 172.21.15.91:0/527513774 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:44.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.220+0000 7f01a6d9d700 1 --2- 172.21.15.91:0/527513774 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01a80d49f0 0x7f01a80b4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.220+0000 7f01a659c700 1 --2- 172.21.15.91:0/527513774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01a80d0870 0x7f01a815a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.220+0000 7f01a5d9b700 1 -- 172.21.15.91:0/527513774 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01a80d49f0 msgr2=0x7f01a80b4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.220+0000 7f01a5d9b700 1 --2- 172.21.15.91:0/527513774 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01a80d49f0 0x7f01a80b4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.220+0000 7f01a5d9b700 1 -- 172.21.15.91:0/527513774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01a80d0870 msgr2=0x7f01a815a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.220+0000 7f01a5d9b700 1 --2- 172.21.15.91:0/527513774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01a80d0870 0x7f01a815a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.220+0000 7f01a5d9b700 1 -- 172.21.15.91:0/527513774 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c0094a0 con 0x7f01a80d0c50 2023-11-24T06:08:44.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.220+0000 7f01a5d9b700 1 --2- 172.21.15.91:0/527513774 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f01a80d0c50 0x7f01a80b13f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f019800c780 tx=0x7f019800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:44.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.220+0000 7f0196ffd700 1 -- 172.21.15.91:0/527513774 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0198004c80 con 0x7f01a80d0c50 2023-11-24T06:08:44.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.220+0000 7f0196ffd700 1 -- 172.21.15.91:0/527513774 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0198004de0 con 0x7f01a80d0c50 2023-11-24T06:08:44.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.220+0000 7f01acac9700 1 -- 172.21.15.91:0/527513774 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a80b3d30 con 0x7f01a80d0c50 2023-11-24T06:08:44.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.220+0000 7f0196ffd700 1 -- 172.21.15.91:0/527513774 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01980072f0 con 0x7f01a80d0c50 2023-11-24T06:08:44.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.220+0000 7f01acac9700 1 -- 172.21.15.91:0/527513774 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a80b3140 con 0x7f01a80d0c50 2023-11-24T06:08:44.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.221+0000 7f01acac9700 1 -- 172.21.15.91:0/527513774 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a80b57c0 con 0x7f01a80d0c50 2023-11-24T06:08:44.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.222+0000 7f0196ffd700 1 -- 172.21.15.91:0/527513774 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0198007450 con 0x7f01a80d0c50 2023-11-24T06:08:44.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.222+0000 7f0196ffd700 1 --2- 172.21.15.91:0/527513774 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f018806aed0 0x7f018806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.222+0000 7f0196ffd700 1 -- 172.21.15.91:0/527513774 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0198089320 con 0x7f01a80d0c50 2023-11-24T06:08:44.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.222+0000 7f01a659c700 1 --2- 172.21.15.91:0/527513774 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f018806aed0 0x7f018806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.222+0000 7f01a659c700 1 --2- 172.21.15.91:0/527513774 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f018806aed0 0x7f018806d390 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f019c0090b0 tx=0x7f019c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:44.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.227+0000 7f0196ffd700 1 -- 172.21.15.91:0/527513774 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01980543e0 con 0x7f01a80d0c50 2023-11-24T06:08:44.419 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.415+0000 7f01acac9700 1 -- 172.21.15.91:0/527513774 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f01a8067a20 con 0x7f01a80d0c50 2023-11-24T06:08:44.419 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.416+0000 7f0196ffd700 1 -- 172.21.15.91:0/527513774 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0198007700 con 0x7f01a80d0c50 2023-11-24T06:08:44.420 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 32 2023-11-24T06:08:44.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.418+0000 7f01acac9700 1 -- 172.21.15.91:0/527513774 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f018806aed0 msgr2=0x7f018806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.418+0000 7f01acac9700 1 --2- 172.21.15.91:0/527513774 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f018806aed0 0x7f018806d390 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f019c0090b0 tx=0x7f019c008030 comp rx=0 tx=0).stop 2023-11-24T06:08:44.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.418+0000 7f01acac9700 1 -- 172.21.15.91:0/527513774 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f01a80d0c50 msgr2=0x7f01a80b13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.418+0000 7f01acac9700 1 --2- 172.21.15.91:0/527513774 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f01a80d0c50 0x7f01a80b13f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f019800c780 tx=0x7f019800ca90 comp rx=0 tx=0).stop 2023-11-24T06:08:44.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.418+0000 7f01acac9700 1 -- 172.21.15.91:0/527513774 shutdown_connections 2023-11-24T06:08:44.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.418+0000 7f01acac9700 1 --2- 172.21.15.91:0/527513774 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f018806aed0 0x7f018806d390 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.418+0000 7f01acac9700 1 --2- 172.21.15.91:0/527513774 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01a80d0870 0x7f01a815a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.418+0000 7f01acac9700 1 --2- 172.21.15.91:0/527513774 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01a80d49f0 0x7f01a80b4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.418+0000 7f01acac9700 1 --2- 172.21.15.91:0/527513774 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f01a80d0c50 0x7f01a80b13f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.418+0000 7f01acac9700 1 -- 172.21.15.91:0/527513774 >> 172.21.15.91:0/527513774 conn(0x7f01a80d3ed0 msgr2=0x7f01a80d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:44.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.418+0000 7f01acac9700 1 -- 172.21.15.91:0/527513774 shutdown_connections 2023-11-24T06:08:44.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.418+0000 7f01acac9700 1 -- 172.21.15.91:0/527513774 wait complete. 2023-11-24T06:08:44.434 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:08:44.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.553+0000 7f1aed66d700 1 Processor -- start 2023-11-24T06:08:44.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.553+0000 7f1aed66d700 1 -- start start 2023-11-24T06:08:44.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1aed66d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1ae80d0870 0x7f1ae80d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1aed66d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f1ae80d0c50 0x7f1ae80d49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1aed66d700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae80b5370 con 0x7f1ae80cf860 2023-11-24T06:08:44.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1aed66d700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae80d2f80 con 0x7f1ae80d0c50 2023-11-24T06:08:44.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1aed66d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80ceae0 con 0x7f1ae80d0870 2023-11-24T06:08:44.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1ae67fc700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f1ae80d0c50 0x7f1ae80d49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.559 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1ae67fc700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f1ae80d0c50 0x7f1ae80d49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59214/0 (socket says 172.21.15.91:59214) 2023-11-24T06:08:44.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1ae67fc700 1 -- 172.21.15.91:0/3367113479 learned_addr learned my addr 172.21.15.91:0/3367113479 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:44.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1ae6ffd700 1 --2- 172.21.15.91:0/3367113479 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1ae80d0870 0x7f1ae80d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1ae67fc700 1 -- 172.21.15.91:0/3367113479 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1ae80d0870 msgr2=0x7f1ae80d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:44.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1ae67fc700 1 --2- 172.21.15.91:0/3367113479 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1ae80d0870 0x7f1ae80d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1ae67fc700 1 -- 172.21.15.91:0/3367113479 >> v1:172.21.15.94:6789/0 conn(0x7f1ae80cf860 legacy=0x7f1ae80b6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1ae67fc700 1 -- 172.21.15.91:0/3367113479 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae80d2ca0 con 0x7f1ae80d0c50 2023-11-24T06:08:44.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1ae67fc700 1 --2- 172.21.15.91:0/3367113479 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1ae80d0c50 0x7f1ae80d49f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0098f0 tx=0x7f1adc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9911c0aed4bda824 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:44.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1ae5ffb700 1 -- 172.21.15.91:0/3367113479 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1adc00b8a0 con 0x7f1ae80d0c50 2023-11-24T06:08:44.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1ae5ffb700 1 -- 172.21.15.91:0/3367113479 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1adc00ba00 con 0x7f1ae80d0c50 2023-11-24T06:08:44.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.554+0000 7f1ae5ffb700 1 -- 172.21.15.91:0/3367113479 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1adc00bd90 con 0x7f1ae80d0c50 2023-11-24T06:08:44.560 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.555+0000 7f1aed66d700 1 -- 172.21.15.91:0/3367113479 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1ae80d0c50 msgr2=0x7f1ae80d49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.555+0000 7f1aed66d700 1 --2- 172.21.15.91:0/3367113479 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1ae80d0c50 0x7f1ae80d49f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0098f0 tx=0x7f1adc009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:44.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.555+0000 7f1aed66d700 1 -- 172.21.15.91:0/3367113479 shutdown_connections 2023-11-24T06:08:44.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.555+0000 7f1aed66d700 1 --2- 172.21.15.91:0/3367113479 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1ae80d0870 0x7f1ae80d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.555+0000 7f1aed66d700 1 --2- 172.21.15.91:0/3367113479 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1ae80d0c50 0x7f1ae80d49f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.555+0000 7f1aed66d700 1 -- 172.21.15.91:0/3367113479 >> 172.21.15.91:0/3367113479 conn(0x7f1ae80d3ed0 msgr2=0x7f1ae80d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:44.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.555+0000 7f1aed66d700 1 -- 172.21.15.91:0/3367113479 shutdown_connections 2023-11-24T06:08:44.561 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.555+0000 7f1aed66d700 1 -- 172.21.15.91:0/3367113479 wait complete. 2023-11-24T06:08:44.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.556+0000 7f1aed66d700 1 Processor -- start 2023-11-24T06:08:44.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.556+0000 7f1aed66d700 1 -- start start 2023-11-24T06:08:44.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.557+0000 7f1aed66d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1ae80cf860 0x7f1ae80b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.557+0000 7f1aed66d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ae80d0870 0x7f1ae80b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.557+0000 7f1aed66d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1ae80d0c50 0x7f1ae814de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.557+0000 7f1aed66d700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80c8630 con 0x7f1ae80d0c50 2023-11-24T06:08:44.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.557+0000 7f1aed66d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80b3c20 con 0x7f1ae80d0870 2023-11-24T06:08:44.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.557+0000 7f1aed66d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80b2220 con 0x7f1ae80cf860 2023-11-24T06:08:44.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.557+0000 7f1ae77fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1ae80d0c50 0x7f1ae814de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.557+0000 7f1ae77fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1ae80d0c50 0x7f1ae814de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59222/0 (socket says 172.21.15.91:59222) 2023-11-24T06:08:44.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.557+0000 7f1ae77fe700 1 -- 172.21.15.91:0/3459555415 learned_addr learned my addr 172.21.15.91:0/3459555415 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:44.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.557+0000 7f1ae6ffd700 1 --2- 172.21.15.91:0/3459555415 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1ae80cf860 0x7f1ae80b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.557+0000 7f1ae67fc700 1 --2- 172.21.15.91:0/3459555415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ae80d0870 0x7f1ae80b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.557+0000 7f1ae77fe700 1 -- 172.21.15.91:0/3459555415 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1ae80cf860 msgr2=0x7f1ae80b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.557+0000 7f1ae77fe700 1 --2- 172.21.15.91:0/3459555415 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1ae80cf860 0x7f1ae80b1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.557+0000 7f1ae77fe700 1 -- 172.21.15.91:0/3459555415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ae80d0870 msgr2=0x7f1ae80b30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.557+0000 7f1ae77fe700 1 --2- 172.21.15.91:0/3459555415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ae80d0870 0x7f1ae80b30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.557+0000 7f1ae77fe700 1 -- 172.21.15.91:0/3459555415 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1adc009550 con 0x7f1ae80d0c50 2023-11-24T06:08:44.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.558+0000 7f1ae77fe700 1 --2- 172.21.15.91:0/3459555415 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1ae80d0c50 0x7f1ae814de70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1ae000cc50 tx=0x7f1ae000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:44.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.558+0000 7f1acf7fe700 1 -- 172.21.15.91:0/3459555415 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0003c00 con 0x7f1ae80d0c50 2023-11-24T06:08:44.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.558+0000 7f1acf7fe700 1 -- 172.21.15.91:0/3459555415 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae0003d60 con 0x7f1ae80d0c50 2023-11-24T06:08:44.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.558+0000 7f1aed66d700 1 -- 172.21.15.91:0/3459555415 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae80b2500 con 0x7f1ae80d0c50 2023-11-24T06:08:44.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.558+0000 7f1acf7fe700 1 -- 172.21.15.91:0/3459555415 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae000aab0 con 0x7f1ae80d0c50 2023-11-24T06:08:44.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.558+0000 7f1aed66d700 1 -- 172.21.15.91:0/3459555415 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae80c8b20 con 0x7f1ae80d0c50 2023-11-24T06:08:44.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.559+0000 7f1aed66d700 1 -- 172.21.15.91:0/3459555415 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae80b57c0 con 0x7f1ae80d0c50 2023-11-24T06:08:44.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.559+0000 7f1acf7fe700 1 -- 172.21.15.91:0/3459555415 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f1ae000ac10 con 0x7f1ae80d0c50 2023-11-24T06:08:44.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.559+0000 7f1acf7fe700 1 --2- 172.21.15.91:0/3459555415 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1ad406aed0 0x7f1ad406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.560+0000 7f1acf7fe700 1 -- 172.21.15.91:0/3459555415 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1ae0089640 con 0x7f1ae80d0c50 2023-11-24T06:08:44.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.560+0000 7f1ae6ffd700 1 --2- 172.21.15.91:0/3459555415 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1ad406aed0 0x7f1ad406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.560+0000 7f1ae6ffd700 1 --2- 172.21.15.91:0/3459555415 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1ad406aed0 0x7f1ad406d390 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8006b20 tx=0x7f1ad8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:44.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.564+0000 7f1acf7fe700 1 -- 172.21.15.91:0/3459555415 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ae0054680 con 0x7f1ae80d0c50 2023-11-24T06:08:44.756 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.752+0000 7f1aed66d700 1 -- 172.21.15.91:0/3459555415 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1ae8067a20 con 0x7f1ae80d0c50 2023-11-24T06:08:44.756 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.753+0000 7f1acf7fe700 1 -- 172.21.15.91:0/3459555415 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1ae0057ca0 con 0x7f1ae80d0c50 2023-11-24T06:08:44.757 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 64 2023-11-24T06:08:44.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.754+0000 7f1aed66d700 1 -- 172.21.15.91:0/3459555415 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1ad406aed0 msgr2=0x7f1ad406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.754+0000 7f1aed66d700 1 --2- 172.21.15.91:0/3459555415 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1ad406aed0 0x7f1ad406d390 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8006b20 tx=0x7f1ad8008040 comp rx=0 tx=0).stop 2023-11-24T06:08:44.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.754+0000 7f1aed66d700 1 -- 172.21.15.91:0/3459555415 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1ae80d0c50 msgr2=0x7f1ae814de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.754+0000 7f1aed66d700 1 --2- 172.21.15.91:0/3459555415 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1ae80d0c50 0x7f1ae814de70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1ae000cc50 tx=0x7f1ae000cf60 comp rx=0 tx=0).stop 2023-11-24T06:08:44.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.755+0000 7f1aed66d700 1 -- 172.21.15.91:0/3459555415 shutdown_connections 2023-11-24T06:08:44.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.755+0000 7f1aed66d700 1 --2- 172.21.15.91:0/3459555415 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1ad406aed0 0x7f1ad406d390 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.755+0000 7f1aed66d700 1 --2- 172.21.15.91:0/3459555415 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1ae80d0c50 0x7f1ae814de70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.755+0000 7f1aed66d700 1 --2- 172.21.15.91:0/3459555415 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1ae80cf860 0x7f1ae80b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.755+0000 7f1aed66d700 1 --2- 172.21.15.91:0/3459555415 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1ae80d0870 0x7f1ae80b30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.755+0000 7f1aed66d700 1 -- 172.21.15.91:0/3459555415 >> 172.21.15.91:0/3459555415 conn(0x7f1ae80d3ed0 msgr2=0x7f1ae806d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:44.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.755+0000 7f1aed66d700 1 -- 172.21.15.91:0/3459555415 shutdown_connections 2023-11-24T06:08:44.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.755+0000 7f1aed66d700 1 -- 172.21.15.91:0/3459555415 wait complete. 2023-11-24T06:08:44.770 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:08:44.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.887+0000 7f8d85a72700 1 Processor -- start 2023-11-24T06:08:44.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.887+0000 7f8d85a72700 1 -- start start 2023-11-24T06:08:44.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.888+0000 7f8d85a72700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d800cf2b0 0x7f8d8015c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.888+0000 7f8d85a72700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f8d800ce5c0 0x7f8d800cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.888+0000 7f8d85a72700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d800d0a90 con 0x7f8d8015bde0 2023-11-24T06:08:44.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.888+0000 7f8d85a72700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d800cefb0 con 0x7f8d800ce5c0 2023-11-24T06:08:44.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.888+0000 7f8d85a72700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8006ee10 con 0x7f8d800cf2b0 2023-11-24T06:08:44.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.888+0000 7f8d7ffff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f8d800ce5c0 0x7f8d800cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.888+0000 7f8d7ffff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f8d800ce5c0 0x7f8d800cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59230/0 (socket says 172.21.15.91:59230) 2023-11-24T06:08:44.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.888+0000 7f8d7f7fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f8d8015bde0 0x7f8d800d01b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54160/0 (socket says 172.21.15.91:54160) 2023-11-24T06:08:44.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.888+0000 7f8d7ffff700 1 -- 172.21.15.91:0/1596834107 learned_addr learned my addr 172.21.15.91:0/1596834107 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:44.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.888+0000 7f8d7effd700 1 --2- 172.21.15.91:0/1596834107 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d800cf2b0 0x7f8d8015c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.888+0000 7f8d7ffff700 1 -- 172.21.15.91:0/1596834107 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d800cf2b0 msgr2=0x7f8d8015c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:44.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.888+0000 7f8d7ffff700 1 --2- 172.21.15.91:0/1596834107 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d800cf2b0 0x7f8d8015c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.888+0000 7f8d7ffff700 1 -- 172.21.15.91:0/1596834107 >> v1:172.21.15.91:6789/0 conn(0x7f8d8015bde0 legacy=0x7f8d800d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.888+0000 7f8d7e7fc700 1 -- 172.21.15.91:0/1596834107 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360634544 0 0) 0x7f8d800d0a90 con 0x7f8d8015bde0 2023-11-24T06:08:44.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.888+0000 7f8d7ffff700 1 -- 172.21.15.91:0/1596834107 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8006f0f0 con 0x7f8d800ce5c0 2023-11-24T06:08:44.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.888+0000 7f8d7effd700 1 --2- 172.21.15.91:0/1596834107 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d800cf2b0 0x7f8d8015c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:44.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.889+0000 7f8d7ffff700 1 --2- 172.21.15.91:0/1596834107 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d800ce5c0 0x7f8d800cea70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8d78009e50 tx=0x7f8d7800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7210d1e214450e93 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:44.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.889+0000 7f8d7e7fc700 1 -- 172.21.15.91:0/1596834107 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7800e070 con 0x7f8d800ce5c0 2023-11-24T06:08:44.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.889+0000 7f8d7e7fc700 1 -- 172.21.15.91:0/1596834107 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7800b9c0 con 0x7f8d800ce5c0 2023-11-24T06:08:44.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.889+0000 7f8d7e7fc700 1 -- 172.21.15.91:0/1596834107 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7800bd70 con 0x7f8d800ce5c0 2023-11-24T06:08:44.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.889+0000 7f8d85a72700 1 -- 172.21.15.91:0/1596834107 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d800ce5c0 msgr2=0x7f8d800cea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.889+0000 7f8d85a72700 1 --2- 172.21.15.91:0/1596834107 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d800ce5c0 0x7f8d800cea70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8d78009e50 tx=0x7f8d7800b330 comp rx=0 tx=0).stop 2023-11-24T06:08:44.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.889+0000 7f8d85a72700 1 -- 172.21.15.91:0/1596834107 shutdown_connections 2023-11-24T06:08:44.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.889+0000 7f8d85a72700 1 --2- 172.21.15.91:0/1596834107 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d800ce5c0 0x7f8d800cea70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.889+0000 7f8d85a72700 1 --2- 172.21.15.91:0/1596834107 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d800cf2b0 0x7f8d8015c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.889+0000 7f8d85a72700 1 -- 172.21.15.91:0/1596834107 >> 172.21.15.91:0/1596834107 conn(0x7f8d800b3720 msgr2=0x7f8d800b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:44.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.889+0000 7f8d85a72700 1 -- 172.21.15.91:0/1596834107 shutdown_connections 2023-11-24T06:08:44.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.889+0000 7f8d85a72700 1 -- 172.21.15.91:0/1596834107 wait complete. 2023-11-24T06:08:44.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.890+0000 7f8d85a72700 1 Processor -- start 2023-11-24T06:08:44.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.891+0000 7f8d85a72700 1 -- start start 2023-11-24T06:08:44.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.891+0000 7f8d85a72700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d800ce5c0 0x7f8d8015c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.891+0000 7f8d85a72700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d800cf2b0 0x7f8d8014c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.891+0000 7f8d85a72700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d8015bde0 0x7f8d8015dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.891+0000 7f8d85a72700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8014c5e0 con 0x7f8d800ce5c0 2023-11-24T06:08:44.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.891+0000 7f8d85a72700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d800b5580 con 0x7f8d8015bde0 2023-11-24T06:08:44.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.891+0000 7f8d85a72700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d800cb550 con 0x7f8d800cf2b0 2023-11-24T06:08:44.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.891+0000 7f8d7f7fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d800ce5c0 0x7f8d8015c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.891+0000 7f8d7f7fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d800ce5c0 0x7f8d8015c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59232/0 (socket says 172.21.15.91:59232) 2023-11-24T06:08:44.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.891+0000 7f8d7f7fe700 1 -- 172.21.15.91:0/2386509883 learned_addr learned my addr 172.21.15.91:0/2386509883 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:44.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.892+0000 7f8d7f7fe700 1 -- 172.21.15.91:0/2386509883 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d800cf2b0 msgr2=0x7f8d8014c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.892+0000 7f8d7f7fe700 1 --2- 172.21.15.91:0/2386509883 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d800cf2b0 0x7f8d8014c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.892+0000 7f8d7f7fe700 1 -- 172.21.15.91:0/2386509883 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d8015bde0 msgr2=0x7f8d8015dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:44.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.892+0000 7f8d7ffff700 1 --2- 172.21.15.91:0/2386509883 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d8015bde0 0x7f8d8015dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.892+0000 7f8d7f7fe700 1 --2- 172.21.15.91:0/2386509883 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d8015bde0 0x7f8d8015dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:44.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.892+0000 7f8d7f7fe700 1 -- 172.21.15.91:0/2386509883 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d78009b30 con 0x7f8d800ce5c0 2023-11-24T06:08:44.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.892+0000 7f8d7f7fe700 1 --2- 172.21.15.91:0/2386509883 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d800ce5c0 0x7f8d8015c540 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8d7400c580 tx=0x7f8d7400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:44.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.892+0000 7f8d67fff700 1 -- 172.21.15.91:0/2386509883 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d74004c80 con 0x7f8d800ce5c0 2023-11-24T06:08:44.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.892+0000 7f8d67fff700 1 -- 172.21.15.91:0/2386509883 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d74004e00 con 0x7f8d800ce5c0 2023-11-24T06:08:44.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.892+0000 7f8d85a72700 1 -- 172.21.15.91:0/2386509883 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d800b4c50 con 0x7f8d800ce5c0 2023-11-24T06:08:44.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.892+0000 7f8d67fff700 1 -- 172.21.15.91:0/2386509883 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d74007310 con 0x7f8d800ce5c0 2023-11-24T06:08:44.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.892+0000 7f8d85a72700 1 -- 172.21.15.91:0/2386509883 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d800b5210 con 0x7f8d800ce5c0 2023-11-24T06:08:44.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.893+0000 7f8d85a72700 1 -- 172.21.15.91:0/2386509883 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d80068670 con 0x7f8d800ce5c0 2023-11-24T06:08:44.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.893+0000 7f8d67fff700 1 -- 172.21.15.91:0/2386509883 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8d74007470 con 0x7f8d800ce5c0 2023-11-24T06:08:44.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.894+0000 7f8d67fff700 1 --2- 172.21.15.91:0/2386509883 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8d6c06aed0 0x7f8d6c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:44.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.894+0000 7f8d7effd700 1 --2- 172.21.15.91:0/2386509883 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8d6c06aed0 0x7f8d6c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:44.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.894+0000 7f8d67fff700 1 -- 172.21.15.91:0/2386509883 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8d74089440 con 0x7f8d800ce5c0 2023-11-24T06:08:44.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.894+0000 7f8d7effd700 1 --2- 172.21.15.91:0/2386509883 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8d6c06aed0 0x7f8d6c06d390 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8d800cb990 tx=0x7f8d70008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:44.902 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:44.898+0000 7f8d67fff700 1 -- 172.21.15.91:0/2386509883 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d740543e0 con 0x7f8d800ce5c0 2023-11-24T06:08:45.135 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.132+0000 7f8d85a72700 1 -- 172.21.15.91:0/2386509883 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d80074950 con 0x7f8d800ce5c0 2023-11-24T06:08:45.136 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.132+0000 7f8d67fff700 1 -- 172.21.15.91:0/2386509883 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4907 (secure 0 0 0) 0x7f8d74057a00 con 0x7f8d800ce5c0 2023-11-24T06:08:45.136 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:45.136 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1329219967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1329219967},{"type":"v1","addr":"172.21.15.94:6834","nonce":1329219967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3439524864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3439524864},{"type":"v1","addr":"172.21.15.94:6835","nonce":3439524864}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6836/3390368219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6834","nonce":3390368219},{"type":"v1","addr":"172.21.15.91:6836","nonce":3390368219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:08:00.798681+0000","modified":"2023-11-24T06:08:04.132764+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.91:6837/1644079645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":1644079645},{"type":"v1","addr":"172.21.15.91:6837","nonce":1644079645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T06:08:45.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.134+0000 7f8d85a72700 1 -- 172.21.15.91:0/2386509883 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8d6c06aed0 msgr2=0x7f8d6c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.134+0000 7f8d85a72700 1 --2- 172.21.15.91:0/2386509883 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8d6c06aed0 0x7f8d6c06d390 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8d800cb990 tx=0x7f8d70008040 comp rx=0 tx=0).stop 2023-11-24T06:08:45.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.134+0000 7f8d85a72700 1 -- 172.21.15.91:0/2386509883 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d800ce5c0 msgr2=0x7f8d8015c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.134+0000 7f8d85a72700 1 --2- 172.21.15.91:0/2386509883 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d800ce5c0 0x7f8d8015c540 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8d7400c580 tx=0x7f8d7400c940 comp rx=0 tx=0).stop 2023-11-24T06:08:45.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.135+0000 7f8d85a72700 1 -- 172.21.15.91:0/2386509883 shutdown_connections 2023-11-24T06:08:45.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.135+0000 7f8d85a72700 1 --2- 172.21.15.91:0/2386509883 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8d6c06aed0 0x7f8d6c06d390 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.135+0000 7f8d85a72700 1 --2- 172.21.15.91:0/2386509883 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8d8015bde0 0x7f8d8015dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.135+0000 7f8d85a72700 1 --2- 172.21.15.91:0/2386509883 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8d800ce5c0 0x7f8d8015c540 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.135+0000 7f8d85a72700 1 --2- 172.21.15.91:0/2386509883 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8d800cf2b0 0x7f8d8014c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.135+0000 7f8d85a72700 1 -- 172.21.15.91:0/2386509883 >> 172.21.15.91:0/2386509883 conn(0x7f8d800b3720 msgr2=0x7f8d80159fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:45.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.135+0000 7f8d85a72700 1 -- 172.21.15.91:0/2386509883 shutdown_connections 2023-11-24T06:08:45.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.135+0000 7f8d85a72700 1 -- 172.21.15.91:0/2386509883 wait complete. 2023-11-24T06:08:45.140 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 8 2023-11-24T06:08:45.150 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:45.270 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.267+0000 7f2821f87700 1 Processor -- start 2023-11-24T06:08:45.271 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.267+0000 7f2821f87700 1 -- start start 2023-11-24T06:08:45.273 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.267+0000 7f2821f87700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f281c0b4150 0x7f281c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.273 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.267+0000 7f2821f87700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f281c0b5ab0 0x7f281c06e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.273 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.268+0000 7f2821f87700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f281c071f50 con 0x7f281c06b620 2023-11-24T06:08:45.273 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.268+0000 7f2821f87700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f281c072230 con 0x7f281c0b5ab0 2023-11-24T06:08:45.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.268+0000 7f2821f87700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281c072510 con 0x7f281c0b4150 2023-11-24T06:08:45.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.268+0000 7f281bfff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f281c0b5ab0 0x7f281c06e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.268+0000 7f281bfff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f281c0b5ab0 0x7f281c06e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59244/0 (socket says 172.21.15.91:59244) 2023-11-24T06:08:45.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.268+0000 7f281bfff700 1 -- 172.21.15.91:0/1692881860 learned_addr learned my addr 172.21.15.91:0/1692881860 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:45.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.268+0000 7f281affd700 1 --2- 172.21.15.91:0/1692881860 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f281c0b4150 0x7f281c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.268+0000 7f281bfff700 1 -- 172.21.15.91:0/1692881860 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f281c0b4150 msgr2=0x7f281c06de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:45.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.268+0000 7f281bfff700 1 --2- 172.21.15.91:0/1692881860 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f281c0b4150 0x7f281c06de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.268+0000 7f281bfff700 1 -- 172.21.15.91:0/1692881860 >> v1:172.21.15.94:6789/0 conn(0x7f281c06b620 legacy=0x7f281c0b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.268+0000 7f281bfff700 1 -- 172.21.15.91:0/1692881860 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f281c0727f0 con 0x7f281c0b5ab0 2023-11-24T06:08:45.274 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.268+0000 7f281affd700 1 --2- 172.21.15.91:0/1692881860 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f281c0b4150 0x7f281c06de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:45.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.268+0000 7f281bfff700 1 --2- 172.21.15.91:0/1692881860 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f281c0b5ab0 0x7f281c06e390 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2814009e50 tx=0x7f281400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=281a3c130453c2ed server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:45.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.268+0000 7f281a7fc700 1 -- 172.21.15.91:0/1692881860 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281400e070 con 0x7f281c0b5ab0 2023-11-24T06:08:45.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.268+0000 7f281a7fc700 1 -- 172.21.15.91:0/1692881860 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281400b9c0 con 0x7f281c0b5ab0 2023-11-24T06:08:45.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.268+0000 7f281a7fc700 1 -- 172.21.15.91:0/1692881860 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281400bd70 con 0x7f281c0b5ab0 2023-11-24T06:08:45.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.269+0000 7f2821f87700 1 -- 172.21.15.91:0/1692881860 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f281c0b5ab0 msgr2=0x7f281c06e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.269+0000 7f2821f87700 1 --2- 172.21.15.91:0/1692881860 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f281c0b5ab0 0x7f281c06e390 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2814009e50 tx=0x7f281400b330 comp rx=0 tx=0).stop 2023-11-24T06:08:45.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.269+0000 7f2821f87700 1 -- 172.21.15.91:0/1692881860 shutdown_connections 2023-11-24T06:08:45.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.269+0000 7f2821f87700 1 --2- 172.21.15.91:0/1692881860 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f281c0b5ab0 0x7f281c06e390 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.269+0000 7f2821f87700 1 --2- 172.21.15.91:0/1692881860 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f281c0b4150 0x7f281c06de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.269+0000 7f2821f87700 1 -- 172.21.15.91:0/1692881860 >> 172.21.15.91:0/1692881860 conn(0x7f281c0b2740 msgr2=0x7f281c0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:45.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.269+0000 7f2821f87700 1 -- 172.21.15.91:0/1692881860 shutdown_connections 2023-11-24T06:08:45.276 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.269+0000 7f2821f87700 1 -- 172.21.15.91:0/1692881860 wait complete. 2023-11-24T06:08:45.278 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.270+0000 7f2821f87700 1 Processor -- start 2023-11-24T06:08:45.278 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.271+0000 7f2821f87700 1 -- start start 2023-11-24T06:08:45.278 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f2821f87700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f281c06b620 0x7f281c1450f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.278 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f2821f87700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c0b4150 0x7f281c0c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f2821f87700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f281c0b5ab0 0x7f281c0ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f2821f87700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c0ca390 con 0x7f281c06b620 2023-11-24T06:08:45.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f2821f87700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f281c0c9710 con 0x7f281c0b4150 2023-11-24T06:08:45.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f2821f87700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f281c0c99f0 con 0x7f281c0b5ab0 2023-11-24T06:08:45.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f281b7fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f281c06b620 0x7f281c1450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f281b7fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f281c06b620 0x7f281c1450f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59246/0 (socket says 172.21.15.91:59246) 2023-11-24T06:08:45.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f281b7fe700 1 -- 172.21.15.91:0/3714706907 learned_addr learned my addr 172.21.15.91:0/3714706907 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:45.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f281affd700 1 --2- 172.21.15.91:0/3714706907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c0b4150 0x7f281c0c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f281bfff700 1 --2- 172.21.15.91:0/3714706907 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f281c0b5ab0 0x7f281c0ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f281b7fe700 1 -- 172.21.15.91:0/3714706907 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f281c0b5ab0 msgr2=0x7f281c0ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f281b7fe700 1 --2- 172.21.15.91:0/3714706907 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f281c0b5ab0 0x7f281c0ae8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f281b7fe700 1 -- 172.21.15.91:0/3714706907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c0b4150 msgr2=0x7f281c0c8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f281b7fe700 1 --2- 172.21.15.91:0/3714706907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c0b4150 0x7f281c0c8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f281b7fe700 1 -- 172.21.15.91:0/3714706907 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2814009b30 con 0x7f281c06b620 2023-11-24T06:08:45.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f281b7fe700 1 --2- 172.21.15.91:0/3714706907 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f281c06b620 0x7f281c1450f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f280c00c450 tx=0x7f280c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:45.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f2803fff700 1 -- 172.21.15.91:0/3714706907 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c0052d0 con 0x7f281c06b620 2023-11-24T06:08:45.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f2803fff700 1 -- 172.21.15.91:0/3714706907 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280c005430 con 0x7f281c06b620 2023-11-24T06:08:45.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f2803fff700 1 -- 172.21.15.91:0/3714706907 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c0038c0 con 0x7f281c06b620 2023-11-24T06:08:45.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f2821f87700 1 -- 172.21.15.91:0/3714706907 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f281c0c9cd0 con 0x7f281c06b620 2023-11-24T06:08:45.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.272+0000 7f2821f87700 1 -- 172.21.15.91:0/3714706907 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f281c0c9370 con 0x7f281c06b620 2023-11-24T06:08:45.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.273+0000 7f2821f87700 1 -- 172.21.15.91:0/3714706907 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f281c0ca6b0 con 0x7f281c06b620 2023-11-24T06:08:45.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.274+0000 7f2803fff700 1 -- 172.21.15.91:0/3714706907 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f280c003a20 con 0x7f281c06b620 2023-11-24T06:08:45.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.274+0000 7f2803fff700 1 --2- 172.21.15.91:0/3714706907 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f280806aed0 0x7f280806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.281 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.274+0000 7f2803fff700 1 -- 172.21.15.91:0/3714706907 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f280c0889b0 con 0x7f281c06b620 2023-11-24T06:08:45.281 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.274+0000 7f281affd700 1 --2- 172.21.15.91:0/3714706907 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f280806aed0 0x7f280806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.281 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.275+0000 7f281affd700 1 --2- 172.21.15.91:0/3714706907 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f280806aed0 0x7f280806d390 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f281c145630 tx=0x7f2810008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:45.282 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.279+0000 7f2803fff700 1 -- 172.21.15.91:0/3714706907 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f280c0539e0 con 0x7f281c06b620 2023-11-24T06:08:45.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.468+0000 7f2821f87700 1 -- 172.21.15.91:0/3714706907 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f281c0ca4d0 con 0x7f281c06b620 2023-11-24T06:08:45.473 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.469+0000 7f2803fff700 1 -- 172.21.15.91:0/3714706907 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16671 (secure 0 0 0) 0x7f280c057000 con 0x7f281c06b620 2023-11-24T06:08:45.473 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:45.473 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":22,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:42.798927+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:45.475 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.471+0000 7f2821f87700 1 -- 172.21.15.91:0/3714706907 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f280806aed0 msgr2=0x7f280806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.475 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.471+0000 7f2821f87700 1 --2- 172.21.15.91:0/3714706907 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f280806aed0 0x7f280806d390 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f281c145630 tx=0x7f2810008040 comp rx=0 tx=0).stop 2023-11-24T06:08:45.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.471+0000 7f2821f87700 1 -- 172.21.15.91:0/3714706907 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f281c06b620 msgr2=0x7f281c1450f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.471+0000 7f2821f87700 1 --2- 172.21.15.91:0/3714706907 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f281c06b620 0x7f281c1450f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f280c00c450 tx=0x7f280c00c810 comp rx=0 tx=0).stop 2023-11-24T06:08:45.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.471+0000 7f2821f87700 1 -- 172.21.15.91:0/3714706907 shutdown_connections 2023-11-24T06:08:45.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.471+0000 7f2821f87700 1 --2- 172.21.15.91:0/3714706907 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f280806aed0 0x7f280806d390 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.471+0000 7f2821f87700 1 --2- 172.21.15.91:0/3714706907 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f281c0b5ab0 0x7f281c0ae8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.471+0000 7f2821f87700 1 --2- 172.21.15.91:0/3714706907 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f281c06b620 0x7f281c1450f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.471+0000 7f2821f87700 1 --2- 172.21.15.91:0/3714706907 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f281c0b4150 0x7f281c0c8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.471+0000 7f2821f87700 1 -- 172.21.15.91:0/3714706907 >> 172.21.15.91:0/3714706907 conn(0x7f281c0b2740 msgr2=0x7f281c145fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:45.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.471+0000 7f2821f87700 1 -- 172.21.15.91:0/3714706907 shutdown_connections 2023-11-24T06:08:45.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.471+0000 7f2821f87700 1 -- 172.21.15.91:0/3714706907 wait complete. 2023-11-24T06:08:45.487 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T06:08:45.487 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:08:45.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.604+0000 7fc797c94700 1 Processor -- start 2023-11-24T06:08:45.608 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.604+0000 7fc797c94700 1 -- start start 2023-11-24T06:08:45.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc797c94700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc79015abb0 0x7fc7900b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc797c94700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fc7900b4d50 0x7fc790071d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc797c94700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79006e550 con 0x7fc7900722d0 2023-11-24T06:08:45.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc797c94700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7900b51d0 con 0x7fc7900b4d50 2023-11-24T06:08:45.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc797c94700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc79006f3b0 con 0x7fc79015abb0 2023-11-24T06:08:45.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc79522f700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fc7900b4d50 0x7fc790071d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc796231700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fc7900722d0 0x7fc79015e760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54190/0 (socket says 172.21.15.91:54190) 2023-11-24T06:08:45.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc79522f700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fc7900b4d50 0x7fc790071d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59258/0 (socket says 172.21.15.91:59258) 2023-11-24T06:08:45.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc796231700 1 -- 172.21.15.91:0/587024767 learned_addr learned my addr 172.21.15.91:0/587024767 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:45.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc795a30700 1 --2- 172.21.15.91:0/587024767 >> v2:172.21.15.94:3300/0 conn(0x7fc79015abb0 0x7fc7900b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc79522f700 1 -- 172.21.15.91:0/587024767 >> v2:172.21.15.94:3300/0 conn(0x7fc79015abb0 msgr2=0x7fc7900b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:45.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc79522f700 1 --2- 172.21.15.91:0/587024767 >> v2:172.21.15.94:3300/0 conn(0x7fc79015abb0 0x7fc7900b4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc79522f700 1 -- 172.21.15.91:0/587024767 >> v1:172.21.15.91:6789/0 conn(0x7fc7900722d0 legacy=0x7fc79015e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc79522f700 1 -- 172.21.15.91:0/587024767 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc790072710 con 0x7fc7900b4d50 2023-11-24T06:08:45.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc794a2e700 1 -- 172.21.15.91:0/587024767 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44831919 0 0) 0x7fc79006e550 con 0x7fc7900722d0 2023-11-24T06:08:45.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc79522f700 1 --2- 172.21.15.91:0/587024767 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7900b4d50 0x7fc790071d90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc790074990 tx=0x7fc780009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c662d897ef10d3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:45.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc794a2e700 1 -- 172.21.15.91:0/587024767 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc780011830 con 0x7fc7900b4d50 2023-11-24T06:08:45.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc794a2e700 1 -- 172.21.15.91:0/587024767 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc780011990 con 0x7fc7900b4d50 2023-11-24T06:08:45.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.605+0000 7fc794a2e700 1 -- 172.21.15.91:0/587024767 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc780011d20 con 0x7fc7900b4d50 2023-11-24T06:08:45.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.606+0000 7fc797c94700 1 -- 172.21.15.91:0/587024767 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7900b4d50 msgr2=0x7fc790071d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.606+0000 7fc797c94700 1 --2- 172.21.15.91:0/587024767 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7900b4d50 0x7fc790071d90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc790074990 tx=0x7fc780009b20 comp rx=0 tx=0).stop 2023-11-24T06:08:45.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.606+0000 7fc797c94700 1 -- 172.21.15.91:0/587024767 shutdown_connections 2023-11-24T06:08:45.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.606+0000 7fc797c94700 1 --2- 172.21.15.91:0/587024767 >> v2:172.21.15.94:3300/0 conn(0x7fc79015abb0 0x7fc7900b4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.606+0000 7fc797c94700 1 --2- 172.21.15.91:0/587024767 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7900b4d50 0x7fc790071d90 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.606+0000 7fc797c94700 1 -- 172.21.15.91:0/587024767 >> 172.21.15.91:0/587024767 conn(0x7fc7900d0aa0 msgr2=0x7fc7900d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:45.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.606+0000 7fc797c94700 1 -- 172.21.15.91:0/587024767 shutdown_connections 2023-11-24T06:08:45.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.606+0000 7fc797c94700 1 -- 172.21.15.91:0/587024767 wait complete. 2023-11-24T06:08:45.617 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.607+0000 7fc797c94700 1 Processor -- start 2023-11-24T06:08:45.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.607+0000 7fc797c94700 1 -- start start 2023-11-24T06:08:45.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc797c94700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7900722d0 0x7fc79006b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc797c94700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79015abb0 0x7fc79015c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc797c94700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc79015cf30 0x7fc79014b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc797c94700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79015a420 con 0x7fc7900722d0 2023-11-24T06:08:45.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc797c94700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79015d550 con 0x7fc79015abb0 2023-11-24T06:08:45.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc797c94700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7900ad9e0 con 0x7fc79015cf30 2023-11-24T06:08:45.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc795a30700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7900722d0 0x7fc79006b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc795a30700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7900722d0 0x7fc79006b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59260/0 (socket says 172.21.15.91:59260) 2023-11-24T06:08:45.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc795a30700 1 -- 172.21.15.91:0/3494009151 learned_addr learned my addr 172.21.15.91:0/3494009151 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:45.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc796231700 1 --2- 172.21.15.91:0/3494009151 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc79015cf30 0x7fc79014b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc79522f700 1 --2- 172.21.15.91:0/3494009151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79015abb0 0x7fc79015c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc795a30700 1 -- 172.21.15.91:0/3494009151 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc79015cf30 msgr2=0x7fc79014b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc795a30700 1 --2- 172.21.15.91:0/3494009151 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc79015cf30 0x7fc79014b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc795a30700 1 -- 172.21.15.91:0/3494009151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79015abb0 msgr2=0x7fc79015c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc795a30700 1 --2- 172.21.15.91:0/3494009151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79015abb0 0x7fc79015c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc795a30700 1 -- 172.21.15.91:0/3494009151 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7800094a0 con 0x7fc7900722d0 2023-11-24T06:08:45.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc796231700 1 --2- 172.21.15.91:0/3494009151 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc79015cf30 0x7fc79014b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:45.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc79522f700 1 --2- 172.21.15.91:0/3494009151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79015abb0 0x7fc79015c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:45.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.608+0000 7fc795a30700 1 --2- 172.21.15.91:0/3494009151 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7900722d0 0x7fc79006b240 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00d800 tx=0x7fc78c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:45.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.609+0000 7fc7867fc700 1 -- 172.21.15.91:0/3494009151 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c0125f0 con 0x7fc7900722d0 2023-11-24T06:08:45.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.609+0000 7fc7867fc700 1 -- 172.21.15.91:0/3494009151 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78c012750 con 0x7fc7900722d0 2023-11-24T06:08:45.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.609+0000 7fc797c94700 1 -- 172.21.15.91:0/3494009151 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7900adcc0 con 0x7fc7900722d0 2023-11-24T06:08:45.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.609+0000 7fc7867fc700 1 -- 172.21.15.91:0/3494009151 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c012b00 con 0x7fc7900722d0 2023-11-24T06:08:45.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.609+0000 7fc797c94700 1 -- 172.21.15.91:0/3494009151 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79006b780 con 0x7fc7900722d0 2023-11-24T06:08:45.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.609+0000 7fc797c94700 1 -- 172.21.15.91:0/3494009151 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc79006b940 con 0x7fc7900722d0 2023-11-24T06:08:45.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.610+0000 7fc7867fc700 1 -- 172.21.15.91:0/3494009151 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc78c012c60 con 0x7fc7900722d0 2023-11-24T06:08:45.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.610+0000 7fc7867fc700 1 --2- 172.21.15.91:0/3494009151 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc77406aed0 0x7fc77406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.610+0000 7fc7867fc700 1 -- 172.21.15.91:0/3494009151 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc78c089970 con 0x7fc7900722d0 2023-11-24T06:08:45.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.610+0000 7fc79522f700 1 --2- 172.21.15.91:0/3494009151 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc77406aed0 0x7fc77406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.611+0000 7fc79522f700 1 --2- 172.21.15.91:0/3494009151 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc77406aed0 0x7fc77406d390 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc790073480 tx=0x7fc780008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:45.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.615+0000 7fc7867fc700 1 -- 172.21.15.91:0/3494009151 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc78c054a30 con 0x7fc7900722d0 2023-11-24T06:08:45.854 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.850+0000 7fc797c94700 1 -- 172.21.15.91:0/3494009151 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc790068670 con 0x7fc7900722d0 2023-11-24T06:08:45.855 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.850+0000 7fc7867fc700 1 -- 172.21.15.91:0/3494009151 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4907 (secure 0 0 0) 0x7fc78c058050 con 0x7fc7900722d0 2023-11-24T06:08:45.855 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:45.855 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1329219967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1329219967},{"type":"v1","addr":"172.21.15.94:6834","nonce":1329219967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3439524864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3439524864},{"type":"v1","addr":"172.21.15.94:6835","nonce":3439524864}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6836/3390368219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6834","nonce":3390368219},{"type":"v1","addr":"172.21.15.91:6836","nonce":3390368219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:08:00.798681+0000","modified":"2023-11-24T06:08:04.132764+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.91:6837/1644079645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":1644079645},{"type":"v1","addr":"172.21.15.91:6837","nonce":1644079645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T06:08:45.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.852+0000 7fc797c94700 1 -- 172.21.15.91:0/3494009151 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc77406aed0 msgr2=0x7fc77406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.852+0000 7fc797c94700 1 --2- 172.21.15.91:0/3494009151 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc77406aed0 0x7fc77406d390 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc790073480 tx=0x7fc780008030 comp rx=0 tx=0).stop 2023-11-24T06:08:45.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.852+0000 7fc797c94700 1 -- 172.21.15.91:0/3494009151 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7900722d0 msgr2=0x7fc79006b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.852+0000 7fc797c94700 1 --2- 172.21.15.91:0/3494009151 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7900722d0 0x7fc79006b240 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00d800 tx=0x7fc78c00dbc0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.852+0000 7fc797c94700 1 -- 172.21.15.91:0/3494009151 shutdown_connections 2023-11-24T06:08:45.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.852+0000 7fc797c94700 1 --2- 172.21.15.91:0/3494009151 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc77406aed0 0x7fc77406d390 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.852+0000 7fc797c94700 1 --2- 172.21.15.91:0/3494009151 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc79015cf30 0x7fc79014b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.852+0000 7fc797c94700 1 --2- 172.21.15.91:0/3494009151 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc7900722d0 0x7fc79006b240 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.852+0000 7fc797c94700 1 --2- 172.21.15.91:0/3494009151 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc79015abb0 0x7fc79015c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.852+0000 7fc797c94700 1 -- 172.21.15.91:0/3494009151 >> 172.21.15.91:0/3494009151 conn(0x7fc7900d0aa0 msgr2=0x7fc7900cfce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:45.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.852+0000 7fc797c94700 1 -- 172.21.15.91:0/3494009151 shutdown_connections 2023-11-24T06:08:45.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.852+0000 7fc797c94700 1 -- 172.21.15.91:0/3494009151 wait complete. 2023-11-24T06:08:45.859 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 8 2023-11-24T06:08:45.868 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:08:45.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.984+0000 7f0c1bdff700 1 Processor -- start 2023-11-24T06:08:45.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.984+0000 7f0c1bdff700 1 -- start start 2023-11-24T06:08:45.990 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.985+0000 7f0c1bdff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c1406b620 0x7f0c140b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.990 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.985+0000 7f0c1bdff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0c1406ee60 0x7f0c140b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.990 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.985+0000 7f0c1bdff700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c14071f50 con 0x7f0c140b5ab0 2023-11-24T06:08:45.990 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.985+0000 7f0c1bdff700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c14072230 con 0x7f0c1406ee60 2023-11-24T06:08:45.991 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.985+0000 7f0c1bdff700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c14072510 con 0x7f0c1406b620 2023-11-24T06:08:45.991 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.985+0000 7f0c19b9b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c1406b620 0x7f0c140b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.991 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.985+0000 7f0c1a39c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0c1406ee60 0x7f0c140b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.991 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.985+0000 7f0c1a39c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0c1406ee60 0x7f0c140b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:43664/0 (socket says 172.21.15.91:43664) 2023-11-24T06:08:45.991 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.985+0000 7f0c1a39c700 1 -- 172.21.15.91:0/3948121688 learned_addr learned my addr 172.21.15.91:0/3948121688 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:45.991 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.985+0000 7f0c19b9b700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c1406b620 0x7f0c140b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:42700/0 (socket says 172.21.15.91:42700) 2023-11-24T06:08:45.991 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.985+0000 7f0c19b9b700 1 -- 172.21.15.91:0/3948121688 >> v2:172.21.15.94:3300/0 conn(0x7f0c1406ee60 msgr2=0x7f0c140b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:45.991 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.985+0000 7f0c19b9b700 1 --2- 172.21.15.91:0/3948121688 >> v2:172.21.15.94:3300/0 conn(0x7f0c1406ee60 0x7f0c140b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.991 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.985+0000 7f0c19b9b700 1 -- 172.21.15.91:0/3948121688 >> v1:172.21.15.94:6789/0 conn(0x7f0c140b5ab0 legacy=0x7f0c14161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.991 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.985+0000 7f0c19b9b700 1 -- 172.21.15.91:0/3948121688 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c140727f0 con 0x7f0c1406b620 2023-11-24T06:08:45.991 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.985+0000 7f0c1a39c700 1 --2- 172.21.15.91:0/3948121688 >> v2:172.21.15.94:3300/0 conn(0x7f0c1406ee60 0x7f0c140b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:08:45.991 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.986+0000 7f0c19b9b700 1 --2- 172.21.15.91:0/3948121688 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c1406b620 0x7f0c140b3a40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0c10009840 tx=0x7f0c10009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9af8f9e63405a90 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:45.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.986+0000 7f0c18b99700 1 -- 172.21.15.91:0/3948121688 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c10011830 con 0x7f0c1406b620 2023-11-24T06:08:45.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.986+0000 7f0c18b99700 1 -- 172.21.15.91:0/3948121688 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c10011990 con 0x7f0c1406b620 2023-11-24T06:08:45.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.986+0000 7f0c18b99700 1 -- 172.21.15.91:0/3948121688 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c10011d20 con 0x7f0c1406b620 2023-11-24T06:08:45.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.986+0000 7f0c1bdff700 1 -- 172.21.15.91:0/3948121688 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c1406b620 msgr2=0x7f0c140b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.986+0000 7f0c1bdff700 1 --2- 172.21.15.91:0/3948121688 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c1406b620 0x7f0c140b3a40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0c10009840 tx=0x7f0c10009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:45.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.986+0000 7f0c1bdff700 1 -- 172.21.15.91:0/3948121688 shutdown_connections 2023-11-24T06:08:45.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.986+0000 7f0c1bdff700 1 --2- 172.21.15.91:0/3948121688 >> v2:172.21.15.94:3300/0 conn(0x7f0c1406ee60 0x7f0c140b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.986+0000 7f0c1bdff700 1 --2- 172.21.15.91:0/3948121688 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c1406b620 0x7f0c140b3a40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.986+0000 7f0c1bdff700 1 -- 172.21.15.91:0/3948121688 >> 172.21.15.91:0/3948121688 conn(0x7f0c140b2740 msgr2=0x7f0c140b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:45.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.986+0000 7f0c1bdff700 1 -- 172.21.15.91:0/3948121688 shutdown_connections 2023-11-24T06:08:45.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.986+0000 7f0c1bdff700 1 -- 172.21.15.91:0/3948121688 wait complete. 2023-11-24T06:08:45.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.988+0000 7f0c1bdff700 1 Processor -- start 2023-11-24T06:08:45.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.988+0000 7f0c1bdff700 1 -- start start 2023-11-24T06:08:45.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.988+0000 7f0c1bdff700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c1406b620 0x7f0c141f7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.988+0000 7f0c1bdff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c1406ee60 0x7f0c141e48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.988+0000 7f0c1bdff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0c140b5ab0 0x7f0c140c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.988+0000 7f0c1bdff700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c140c7f40 con 0x7f0c140b5ab0 2023-11-24T06:08:45.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.988+0000 7f0c1bdff700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c141f7f80 con 0x7f0c1406b620 2023-11-24T06:08:45.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.988+0000 7f0c1bdff700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c141f76e0 con 0x7f0c1406ee60 2023-11-24T06:08:45.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.988+0000 7f0c1a39c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0c140b5ab0 0x7f0c140c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.988+0000 7f0c1a39c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0c140b5ab0 0x7f0c140c7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59278/0 (socket says 172.21.15.91:59278) 2023-11-24T06:08:45.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.988+0000 7f0c1a39c700 1 -- 172.21.15.91:0/3324227254 learned_addr learned my addr 172.21.15.91:0/3324227254 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:45.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.989+0000 7f0c19b9b700 1 --2- 172.21.15.91:0/3324227254 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c1406b620 0x7f0c141f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.989+0000 7f0c1939a700 1 --2- 172.21.15.91:0/3324227254 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c1406ee60 0x7f0c141e48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.989+0000 7f0c1a39c700 1 -- 172.21.15.91:0/3324227254 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c1406ee60 msgr2=0x7f0c141e48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.989+0000 7f0c1a39c700 1 --2- 172.21.15.91:0/3324227254 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c1406ee60 0x7f0c141e48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.989+0000 7f0c1a39c700 1 -- 172.21.15.91:0/3324227254 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c1406b620 msgr2=0x7f0c141f7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:45.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.989+0000 7f0c1a39c700 1 --2- 172.21.15.91:0/3324227254 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c1406b620 0x7f0c141f7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:45.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.989+0000 7f0c1a39c700 1 -- 172.21.15.91:0/3324227254 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c009b30 con 0x7f0c140b5ab0 2023-11-24T06:08:45.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.989+0000 7f0c19b9b700 1 --2- 172.21.15.91:0/3324227254 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c1406b620 0x7f0c141f7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:45.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.989+0000 7f0c1a39c700 1 --2- 172.21.15.91:0/3324227254 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0c140b5ab0 0x7f0c140c7a00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c00efc0 tx=0x7f0c0c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:45.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.989+0000 7f0c0a7fc700 1 -- 172.21.15.91:0/3324227254 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c00cbf0 con 0x7f0c140b5ab0 2023-11-24T06:08:45.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.989+0000 7f0c0a7fc700 1 -- 172.21.15.91:0/3324227254 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0c00f040 con 0x7f0c140b5ab0 2023-11-24T06:08:45.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.989+0000 7f0c0a7fc700 1 -- 172.21.15.91:0/3324227254 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c014480 con 0x7f0c140b5ab0 2023-11-24T06:08:45.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.989+0000 7f0c1bdff700 1 -- 172.21.15.91:0/3324227254 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c100094a0 con 0x7f0c140b5ab0 2023-11-24T06:08:45.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.989+0000 7f0c1bdff700 1 -- 172.21.15.91:0/3324227254 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c141f7ca0 con 0x7f0c140b5ab0 2023-11-24T06:08:45.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.990+0000 7f0c1bdff700 1 -- 172.21.15.91:0/3324227254 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c14068670 con 0x7f0c140b5ab0 2023-11-24T06:08:45.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.991+0000 7f0c0a7fc700 1 -- 172.21.15.91:0/3324227254 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0c0c0145e0 con 0x7f0c140b5ab0 2023-11-24T06:08:45.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.991+0000 7f0c0a7fc700 1 --2- 172.21.15.91:0/3324227254 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0bf806aed0 0x7f0bf806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:45.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.991+0000 7f0c0a7fc700 1 -- 172.21.15.91:0/3324227254 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0c0c089cb0 con 0x7f0c140b5ab0 2023-11-24T06:08:45.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.991+0000 7f0c19b9b700 1 --2- 172.21.15.91:0/3324227254 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0bf806aed0 0x7f0bf806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:45.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.991+0000 7f0c19b9b700 1 --2- 172.21.15.91:0/3324227254 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0bf806aed0 0x7f0bf806d390 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0c10000c00 tx=0x7f0c10008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:45.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:45.996+0000 7f0c0a7fc700 1 -- 172.21.15.91:0/3324227254 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0c0c054d70 con 0x7f0c140b5ab0 2023-11-24T06:08:46.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.230+0000 7f0c1bdff700 1 -- 172.21.15.91:0/3324227254 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c1406bd40 con 0x7f0c140b5ab0 2023-11-24T06:08:46.235 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.231+0000 7f0c0a7fc700 1 -- 172.21.15.91:0/3324227254 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4907 (secure 0 0 0) 0x7f0c0c058390 con 0x7f0c140b5ab0 2023-11-24T06:08:46.235 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:46.235 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1329219967","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1329219967},{"type":"v1","addr":"172.21.15.94:6834","nonce":1329219967}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4261,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/3439524864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":3439524864},{"type":"v1","addr":"172.21.15.94:6835","nonce":3439524864}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6836/3390368219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6834","nonce":3390368219},{"type":"v1","addr":"172.21.15.91:6836","nonce":3390368219}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:08:00.798681+0000","modified":"2023-11-24T06:08:04.132764+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4290":{"gid":4290,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.91:6837/1644079645","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":1644079645},{"type":"v1","addr":"172.21.15.91:6837","nonce":1644079645}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-11-24T06:08:46.237 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.232+0000 7f0c1bdff700 1 -- 172.21.15.91:0/3324227254 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0bf806aed0 msgr2=0x7f0bf806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:46.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.232+0000 7f0c1bdff700 1 --2- 172.21.15.91:0/3324227254 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0bf806aed0 0x7f0bf806d390 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0c10000c00 tx=0x7f0c10008030 comp rx=0 tx=0).stop 2023-11-24T06:08:46.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.232+0000 7f0c1bdff700 1 -- 172.21.15.91:0/3324227254 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0c140b5ab0 msgr2=0x7f0c140c7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:46.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.232+0000 7f0c1bdff700 1 --2- 172.21.15.91:0/3324227254 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0c140b5ab0 0x7f0c140c7a00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c00efc0 tx=0x7f0c0c00c3a0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.233+0000 7f0c1bdff700 1 -- 172.21.15.91:0/3324227254 shutdown_connections 2023-11-24T06:08:46.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.233+0000 7f0c1bdff700 1 --2- 172.21.15.91:0/3324227254 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0bf806aed0 0x7f0bf806d390 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.233+0000 7f0c1bdff700 1 --2- 172.21.15.91:0/3324227254 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c1406b620 0x7f0c141f7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.233+0000 7f0c1bdff700 1 --2- 172.21.15.91:0/3324227254 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0c140b5ab0 0x7f0c140c7a00 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.233+0000 7f0c1bdff700 1 --2- 172.21.15.91:0/3324227254 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c1406ee60 0x7f0c141e48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.233+0000 7f0c1bdff700 1 -- 172.21.15.91:0/3324227254 >> 172.21.15.91:0/3324227254 conn(0x7f0c140b2740 msgr2=0x7f0c140b06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:46.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.233+0000 7f0c1bdff700 1 -- 172.21.15.91:0/3324227254 shutdown_connections 2023-11-24T06:08:46.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.233+0000 7f0c1bdff700 1 -- 172.21.15.91:0/3324227254 wait complete. 2023-11-24T06:08:46.240 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 8 2023-11-24T06:08:46.248 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:46.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.364+0000 7fcd5cd24700 1 Processor -- start 2023-11-24T06:08:46.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.364+0000 7fcd5cd24700 1 -- start start 2023-11-24T06:08:46.371 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.365+0000 7fcd5cd24700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcd580b4150 0x7fcd5806de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:46.371 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.365+0000 7fcd5cd24700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fcd580b5ab0 0x7fcd5806e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:46.371 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.365+0000 7fcd5cd24700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd58071f50 con 0x7fcd5806b620 2023-11-24T06:08:46.371 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.365+0000 7fcd5cd24700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd58072230 con 0x7fcd580b5ab0 2023-11-24T06:08:46.371 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.365+0000 7fcd5cd24700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd58072510 con 0x7fcd580b4150 2023-11-24T06:08:46.371 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.366+0000 7fcd5659c700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fcd5806b620 0x7fcd580b3a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54216/0 (socket says 172.21.15.91:54216) 2023-11-24T06:08:46.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.366+0000 7fcd5659c700 1 -- 172.21.15.91:0/2850606479 learned_addr learned my addr 172.21.15.91:0/2850606479 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:46.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.366+0000 7fcd56d9d700 1 --2- 172.21.15.91:0/2850606479 >> v2:172.21.15.94:3300/0 conn(0x7fcd580b5ab0 0x7fcd5806e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:46.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.366+0000 7fcd55d9b700 1 --2- 172.21.15.91:0/2850606479 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcd580b4150 0x7fcd5806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:46.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.366+0000 7fcd5559a700 1 -- 172.21.15.91:0/2850606479 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479801328 0 0) 0x7fcd58071f50 con 0x7fcd5806b620 2023-11-24T06:08:46.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.366+0000 7fcd5559a700 1 -- 172.21.15.91:0/2850606479 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd400035b0 con 0x7fcd5806b620 2023-11-24T06:08:46.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.366+0000 7fcd56d9d700 1 -- 172.21.15.91:0/2850606479 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcd580b4150 msgr2=0x7fcd5806de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:46.372 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.366+0000 7fcd56d9d700 1 --2- 172.21.15.91:0/2850606479 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcd580b4150 0x7fcd5806de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.366+0000 7fcd56d9d700 1 -- 172.21.15.91:0/2850606479 >> v1:172.21.15.91:6789/0 conn(0x7fcd5806b620 legacy=0x7fcd580b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:46.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.366+0000 7fcd56d9d700 1 -- 172.21.15.91:0/2850606479 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd580727f0 con 0x7fcd580b5ab0 2023-11-24T06:08:46.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.366+0000 7fcd55d9b700 1 --2- 172.21.15.91:0/2850606479 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcd580b4150 0x7fcd5806de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:08:46.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.366+0000 7fcd56d9d700 1 --2- 172.21.15.91:0/2850606479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd580b5ab0 0x7fcd5806e390 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcd50009d90 tx=0x7fcd5000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1b8dc5ed837c87c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:46.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.366+0000 7fcd5559a700 1 -- 172.21.15.91:0/2850606479 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5000e070 con 0x7fcd580b5ab0 2023-11-24T06:08:46.373 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.366+0000 7fcd5559a700 1 -- 172.21.15.91:0/2850606479 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5000bb20 con 0x7fcd580b5ab0 2023-11-24T06:08:46.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.366+0000 7fcd5559a700 1 -- 172.21.15.91:0/2850606479 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5000be90 con 0x7fcd580b5ab0 2023-11-24T06:08:46.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.367+0000 7fcd5cd24700 1 -- 172.21.15.91:0/2850606479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd580b5ab0 msgr2=0x7fcd5806e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:46.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.367+0000 7fcd5cd24700 1 --2- 172.21.15.91:0/2850606479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd580b5ab0 0x7fcd5806e390 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcd50009d90 tx=0x7fcd5000b330 comp rx=0 tx=0).stop 2023-11-24T06:08:46.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.367+0000 7fcd5cd24700 1 -- 172.21.15.91:0/2850606479 shutdown_connections 2023-11-24T06:08:46.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.367+0000 7fcd5cd24700 1 --2- 172.21.15.91:0/2850606479 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd580b5ab0 0x7fcd5806e390 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.374 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.367+0000 7fcd5cd24700 1 --2- 172.21.15.91:0/2850606479 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcd580b4150 0x7fcd5806de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.375 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.367+0000 7fcd5cd24700 1 -- 172.21.15.91:0/2850606479 >> 172.21.15.91:0/2850606479 conn(0x7fcd580b2740 msgr2=0x7fcd580b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:46.375 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.367+0000 7fcd5cd24700 1 -- 172.21.15.91:0/2850606479 shutdown_connections 2023-11-24T06:08:46.375 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.367+0000 7fcd5cd24700 1 -- 172.21.15.91:0/2850606479 wait complete. 2023-11-24T06:08:46.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.369+0000 7fcd5cd24700 1 Processor -- start 2023-11-24T06:08:46.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.369+0000 7fcd5cd24700 1 -- start start 2023-11-24T06:08:46.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.369+0000 7fcd5cd24700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd5806b620 0x7fcd581444a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:46.377 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd5cd24700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcd580b4150 0x7fcd581def80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:46.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd5cd24700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcd580b5ab0 0x7fcd5806e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:46.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd5cd24700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd581dd720 con 0x7fcd580b4150 2023-11-24T06:08:46.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd5cd24700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd581dda00 con 0x7fcd5806b620 2023-11-24T06:08:46.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd5cd24700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd581ddce0 con 0x7fcd580b5ab0 2023-11-24T06:08:46.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd55d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcd580b4150 0x7fcd581def80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:46.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd55d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcd580b4150 0x7fcd581def80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59290/0 (socket says 172.21.15.91:59290) 2023-11-24T06:08:46.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd55d9b700 1 -- 172.21.15.91:0/1608955434 learned_addr learned my addr 172.21.15.91:0/1608955434 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:46.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd55d9b700 1 -- 172.21.15.91:0/1608955434 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcd580b5ab0 msgr2=0x7fcd5806e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:46.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd55d9b700 1 --2- 172.21.15.91:0/1608955434 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcd580b5ab0 0x7fcd5806e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd55d9b700 1 -- 172.21.15.91:0/1608955434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd5806b620 msgr2=0x7fcd581444a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:46.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd5659c700 1 --2- 172.21.15.91:0/1608955434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd5806b620 0x7fcd581444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:46.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd55d9b700 1 --2- 172.21.15.91:0/1608955434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd5806b620 0x7fcd581444a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd55d9b700 1 -- 172.21.15.91:0/1608955434 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd480094a0 con 0x7fcd580b4150 2023-11-24T06:08:46.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd55d9b700 1 --2- 172.21.15.91:0/1608955434 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcd580b4150 0x7fcd581def80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800ebe0 tx=0x7fcd48009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:46.380 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd46ffd700 1 -- 172.21.15.91:0/1608955434 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4800c910 con 0x7fcd580b4150 2023-11-24T06:08:46.380 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd46ffd700 1 -- 172.21.15.91:0/1608955434 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd4800ca70 con 0x7fcd580b4150 2023-11-24T06:08:46.380 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd46ffd700 1 -- 172.21.15.91:0/1608955434 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4800cde0 con 0x7fcd580b4150 2023-11-24T06:08:46.380 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd5cd24700 1 -- 172.21.15.91:0/1608955434 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd50009a70 con 0x7fcd580b4150 2023-11-24T06:08:46.380 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.370+0000 7fcd5cd24700 1 -- 172.21.15.91:0/1608955434 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd581e1550 con 0x7fcd580b4150 2023-11-24T06:08:46.380 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.371+0000 7fcd5cd24700 1 -- 172.21.15.91:0/1608955434 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd581e1770 con 0x7fcd580b4150 2023-11-24T06:08:46.380 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.372+0000 7fcd46ffd700 1 -- 172.21.15.91:0/1608955434 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcd48013470 con 0x7fcd580b4150 2023-11-24T06:08:46.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.372+0000 7fcd46ffd700 1 --2- 172.21.15.91:0/1608955434 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcd3806aed0 0x7fcd3806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:46.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.372+0000 7fcd46ffd700 1 -- 172.21.15.91:0/1608955434 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcd48089770 con 0x7fcd580b4150 2023-11-24T06:08:46.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.372+0000 7fcd5659c700 1 --2- 172.21.15.91:0/1608955434 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcd3806aed0 0x7fcd3806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:46.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.373+0000 7fcd5659c700 1 --2- 172.21.15.91:0/1608955434 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcd3806aed0 0x7fcd3806d390 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c000f60 tx=0x7fcd4c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:46.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.376+0000 7fcd46ffd700 1 -- 172.21.15.91:0/1608955434 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd581e1770 con 0x7fcd580b4150 2023-11-24T06:08:46.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.568+0000 7fcd5cd24700 1 -- 172.21.15.91:0/1608955434 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd581e1770 con 0x7fcd580b4150 2023-11-24T06:08:46.573 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.569+0000 7fcd46ffd700 1 -- 172.21.15.91:0/1608955434 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16671 (secure 0 0 0) 0x7fcd48054830 con 0x7fcd580b4150 2023-11-24T06:08:46.573 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:46.573 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":22,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:42.798927+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:07:58.710106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-11-24T06:07:59.774819+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"3.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:46.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.571+0000 7fcd5cd24700 1 -- 172.21.15.91:0/1608955434 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcd3806aed0 msgr2=0x7fcd3806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:46.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.571+0000 7fcd5cd24700 1 --2- 172.21.15.91:0/1608955434 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcd3806aed0 0x7fcd3806d390 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c000f60 tx=0x7fcd4c009040 comp rx=0 tx=0).stop 2023-11-24T06:08:46.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.571+0000 7fcd5cd24700 1 -- 172.21.15.91:0/1608955434 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcd580b4150 msgr2=0x7fcd581def80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:46.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.571+0000 7fcd5cd24700 1 --2- 172.21.15.91:0/1608955434 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcd580b4150 0x7fcd581def80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800ebe0 tx=0x7fcd48009ec0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.571+0000 7fcd5cd24700 1 -- 172.21.15.91:0/1608955434 shutdown_connections 2023-11-24T06:08:46.576 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.571+0000 7fcd5cd24700 1 --2- 172.21.15.91:0/1608955434 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcd3806aed0 0x7fcd3806d390 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.577 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.571+0000 7fcd5cd24700 1 --2- 172.21.15.91:0/1608955434 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcd5806b620 0x7fcd581444a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.577 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.571+0000 7fcd5cd24700 1 --2- 172.21.15.91:0/1608955434 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcd580b5ab0 0x7fcd5806e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.577 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.571+0000 7fcd5cd24700 1 --2- 172.21.15.91:0/1608955434 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcd580b4150 0x7fcd581def80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.577 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.571+0000 7fcd5cd24700 1 -- 172.21.15.91:0/1608955434 >> 172.21.15.91:0/1608955434 conn(0x7fcd580b2740 msgr2=0x7fcd580b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:46.577 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.571+0000 7fcd5cd24700 1 -- 172.21.15.91:0/1608955434 shutdown_connections 2023-11-24T06:08:46.577 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.571+0000 7fcd5cd24700 1 -- 172.21.15.91:0/1608955434 wait complete. 2023-11-24T06:08:46.588 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T06:08:46.710 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.705+0000 7f42f25e8700 1 Processor -- start 2023-11-24T06:08:46.710 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.706+0000 7f42f25e8700 1 -- start start 2023-11-24T06:08:46.717 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.706+0000 7f42f25e8700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f42ec15e1c0 0x7f42ec06eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:46.717 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.706+0000 7f42f25e8700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f42ec06fc50 0x7f42ec06ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:46.717 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.706+0000 7f42f25e8700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42ec0d1230 con 0x7f42ec0b0e90 2023-11-24T06:08:46.718 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.706+0000 7f42f25e8700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42ec0d0c70 con 0x7f42ec15e1c0 2023-11-24T06:08:46.718 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.706+0000 7f42f25e8700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42ec0d0f50 con 0x7f42ec06fc50 2023-11-24T06:08:46.718 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.706+0000 7f42ebfff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f42ec15e1c0 0x7f42ec06eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:46.718 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.706+0000 7f42eb7fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f42ec0b0e90 0x7f42ec06f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54232/0 (socket says 172.21.15.91:54232) 2023-11-24T06:08:46.718 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.706+0000 7f42eb7fe700 1 -- 172.21.15.91:0/2972463343 learned_addr learned my addr 172.21.15.91:0/2972463343 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:46.718 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.707+0000 7f42ebfff700 1 -- 172.21.15.91:0/2972463343 >> v2:172.21.15.94:3300/0 conn(0x7f42ec06fc50 msgr2=0x7f42ec06ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:46.719 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.707+0000 7f42ebfff700 1 --2- 172.21.15.91:0/2972463343 >> v2:172.21.15.94:3300/0 conn(0x7f42ec06fc50 0x7f42ec06ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.719 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.707+0000 7f42ebfff700 1 -- 172.21.15.91:0/2972463343 >> v1:172.21.15.91:6789/0 conn(0x7f42ec0b0e90 legacy=0x7f42ec06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:46.719 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.707+0000 7f42ebfff700 1 -- 172.21.15.91:0/2972463343 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec06c210 con 0x7f42ec15e1c0 2023-11-24T06:08:46.719 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.707+0000 7f42eaffd700 1 -- 172.21.15.91:0/2972463343 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525471864 0 0) 0x7f42ec0d1230 con 0x7f42ec0b0e90 2023-11-24T06:08:46.719 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.707+0000 7f42ebfff700 1 --2- 172.21.15.91:0/2972463343 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec15e1c0 0x7f42ec06eef0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f42ec162e30 tx=0x7f42e0009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2985300a2815aec7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:46.719 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.707+0000 7f42eaffd700 1 -- 172.21.15.91:0/2972463343 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e0011830 con 0x7f42ec15e1c0 2023-11-24T06:08:46.719 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.707+0000 7f42eaffd700 1 -- 172.21.15.91:0/2972463343 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e0011990 con 0x7f42ec15e1c0 2023-11-24T06:08:46.720 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.707+0000 7f42eaffd700 1 -- 172.21.15.91:0/2972463343 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e0011d20 con 0x7f42ec15e1c0 2023-11-24T06:08:46.720 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.708+0000 7f42f25e8700 1 -- 172.21.15.91:0/2972463343 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec15e1c0 msgr2=0x7f42ec06eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:46.720 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.708+0000 7f42f25e8700 1 --2- 172.21.15.91:0/2972463343 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec15e1c0 0x7f42ec06eef0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f42ec162e30 tx=0x7f42e0009b20 comp rx=0 tx=0).stop 2023-11-24T06:08:46.720 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.708+0000 7f42f25e8700 1 -- 172.21.15.91:0/2972463343 shutdown_connections 2023-11-24T06:08:46.720 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.708+0000 7f42f25e8700 1 --2- 172.21.15.91:0/2972463343 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec15e1c0 0x7f42ec06eef0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.720 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.708+0000 7f42f25e8700 1 --2- 172.21.15.91:0/2972463343 >> v2:172.21.15.94:3300/0 conn(0x7f42ec06fc50 0x7f42ec06ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.720 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.708+0000 7f42f25e8700 1 -- 172.21.15.91:0/2972463343 >> 172.21.15.91:0/2972463343 conn(0x7f42ec0b3720 msgr2=0x7f42ec0b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:46.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.708+0000 7f42f25e8700 1 -- 172.21.15.91:0/2972463343 shutdown_connections 2023-11-24T06:08:46.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.708+0000 7f42f25e8700 1 -- 172.21.15.91:0/2972463343 wait complete. 2023-11-24T06:08:46.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.709+0000 7f42f25e8700 1 Processor -- start 2023-11-24T06:08:46.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.709+0000 7f42f25e8700 1 -- start start 2023-11-24T06:08:46.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42f25e8700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec06fc50 0x7f42ec1f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:46.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42f25e8700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42ec0b0e90 0x7f42ec148310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:46.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42f25e8700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42ec15e1c0 0x7f42ec145bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:46.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42f25e8700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec1f4c60 con 0x7f42ec06fc50 2023-11-24T06:08:46.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42f25e8700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec1f43c0 con 0x7f42ec0b0e90 2023-11-24T06:08:46.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42f25e8700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec1f46a0 con 0x7f42ec15e1c0 2023-11-24T06:08:46.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42ebfff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec06fc50 0x7f42ec1f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:46.722 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42ebfff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec06fc50 0x7f42ec1f3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59302/0 (socket says 172.21.15.91:59302) 2023-11-24T06:08:46.723 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42ebfff700 1 -- 172.21.15.91:0/1575446838 learned_addr learned my addr 172.21.15.91:0/1575446838 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:46.723 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42ebfff700 1 -- 172.21.15.91:0/1575446838 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42ec15e1c0 msgr2=0x7f42ec145bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:46.723 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42eb7fe700 1 --2- 172.21.15.91:0/1575446838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42ec0b0e90 0x7f42ec148310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:46.723 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42ebfff700 1 --2- 172.21.15.91:0/1575446838 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42ec15e1c0 0x7f42ec145bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.723 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42ebfff700 1 -- 172.21.15.91:0/1575446838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42ec0b0e90 msgr2=0x7f42ec148310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:46.723 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42ebfff700 1 --2- 172.21.15.91:0/1575446838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42ec0b0e90 0x7f42ec148310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.723 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42ebfff700 1 -- 172.21.15.91:0/1575446838 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e00094a0 con 0x7f42ec06fc50 2023-11-24T06:08:46.724 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42ebfff700 1 --2- 172.21.15.91:0/1575446838 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec06fc50 0x7f42ec1f3cc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f42e0014040 tx=0x7f42e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:46.724 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42e8ff9700 1 -- 172.21.15.91:0/1575446838 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e00038a0 con 0x7f42ec06fc50 2023-11-24T06:08:46.724 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42e8ff9700 1 -- 172.21.15.91:0/1575446838 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e001e070 con 0x7f42ec06fc50 2023-11-24T06:08:46.724 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42e8ff9700 1 -- 172.21.15.91:0/1575446838 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e0003c60 con 0x7f42ec06fc50 2023-11-24T06:08:46.724 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.710+0000 7f42f25e8700 1 -- 172.21.15.91:0/1575446838 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42ec1f4980 con 0x7f42ec06fc50 2023-11-24T06:08:46.724 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.711+0000 7f42f25e8700 1 -- 172.21.15.91:0/1575446838 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42ec148850 con 0x7f42ec06fc50 2023-11-24T06:08:46.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.711+0000 7f42f25e8700 1 -- 172.21.15.91:0/1575446838 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42ec147c20 con 0x7f42ec06fc50 2023-11-24T06:08:46.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.712+0000 7f42e8ff9700 1 -- 172.21.15.91:0/1575446838 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f42e002d450 con 0x7f42ec06fc50 2023-11-24T06:08:46.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.712+0000 7f42e8ff9700 1 --2- 172.21.15.91:0/1575446838 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f42d806aed0 0x7f42d806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:46.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.712+0000 7f42e8ff9700 1 -- 172.21.15.91:0/1575446838 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f42e008a760 con 0x7f42ec06fc50 2023-11-24T06:08:46.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.712+0000 7f42eb7fe700 1 --2- 172.21.15.91:0/1575446838 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f42d806aed0 0x7f42d806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:46.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.713+0000 7f42eb7fe700 1 --2- 172.21.15.91:0/1575446838 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f42d806aed0 0x7f42d806d390 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f42dc000f60 tx=0x7f42dc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:46.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.717+0000 7f42e8ff9700 1 -- 172.21.15.91:0/1575446838 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42e0055820 con 0x7f42ec06fc50 2023-11-24T06:08:46.957 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.953+0000 7f42f25e8700 1 -- 172.21.15.91:0/1575446838 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f42ec06ced0 con 0x7f42ec06fc50 2023-11-24T06:08:46.957 INFO:tasks.ceph.mon.a.smithi091.stderr:2023-11-24T06:08:46.954+0000 7f8e5603a700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T06:08:46.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.955+0000 7f42e8ff9700 1 -- 172.21.15.91:0/1575446838 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f42e0020090 con 0x7f42ec06fc50 2023-11-24T06:08:46.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.958+0000 7f42f25e8700 1 -- 172.21.15.91:0/1575446838 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f42d806aed0 msgr2=0x7f42d806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:46.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.958+0000 7f42f25e8700 1 --2- 172.21.15.91:0/1575446838 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f42d806aed0 0x7f42d806d390 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f42dc000f60 tx=0x7f42dc009040 comp rx=0 tx=0).stop 2023-11-24T06:08:46.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.958+0000 7f42f25e8700 1 -- 172.21.15.91:0/1575446838 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec06fc50 msgr2=0x7f42ec1f3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:46.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.958+0000 7f42f25e8700 1 --2- 172.21.15.91:0/1575446838 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec06fc50 0x7f42ec1f3cc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f42e0014040 tx=0x7f42e0004750 comp rx=0 tx=0).stop 2023-11-24T06:08:46.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.958+0000 7f42f25e8700 1 -- 172.21.15.91:0/1575446838 shutdown_connections 2023-11-24T06:08:46.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.958+0000 7f42f25e8700 1 --2- 172.21.15.91:0/1575446838 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f42d806aed0 0x7f42d806d390 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.958+0000 7f42f25e8700 1 --2- 172.21.15.91:0/1575446838 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f42ec15e1c0 0x7f42ec145bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.958+0000 7f42f25e8700 1 --2- 172.21.15.91:0/1575446838 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f42ec06fc50 0x7f42ec1f3cc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.958+0000 7f42f25e8700 1 --2- 172.21.15.91:0/1575446838 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f42ec0b0e90 0x7f42ec148310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:46.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.958+0000 7f42f25e8700 1 -- 172.21.15.91:0/1575446838 >> 172.21.15.91:0/1575446838 conn(0x7f42ec0b3720 msgr2=0x7f42ec0b3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:46.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.958+0000 7f42f25e8700 1 -- 172.21.15.91:0/1575446838 shutdown_connections 2023-11-24T06:08:46.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:46.958+0000 7f42f25e8700 1 -- 172.21.15.91:0/1575446838 wait complete. 2023-11-24T06:08:46.963 INFO:teuthology.orchestra.run.smithi091.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T06:08:46.975 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T06:08:46.999 INFO:tasks.ceph.mds.a.smithi091.stdout:starting mds.a at 2023-11-24T06:08:47.097 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.093+0000 7fe5f0abb700 1 Processor -- start 2023-11-24T06:08:47.097 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.093+0000 7fe5f0abb700 1 -- start start 2023-11-24T06:08:47.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.094+0000 7fe5f0abb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5ec1440f0 0x7fe5ec0b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:47.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.094+0000 7fe5f0abb700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ec143ec0 con 0x7fe5ec143ae0 2023-11-24T06:08:47.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.094+0000 7fe5f0abb700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ec0b3ae0 con 0x7fe5ec0722c0 2023-11-24T06:08:47.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.094+0000 7fe5f0abb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec0b40a0 con 0x7fe5ec1440f0 2023-11-24T06:08:47.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.095+0000 7fe5ea59c700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fe5ec143ae0 0x7fe5ec071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54256/0 (socket says 172.21.15.91:54256) 2023-11-24T06:08:47.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.095+0000 7fe5ea59c700 1 -- 172.21.15.91:0/385684116 learned_addr learned my addr 172.21.15.91:0/385684116 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:47.101 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.095+0000 7fe5ead9d700 1 --2- 172.21.15.91:0/385684116 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5ec1440f0 0x7fe5ec0b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:47.101 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.095+0000 7fe5e959a700 1 -- 172.21.15.91:0/385684116 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308137874 0 0) 0x7fe5ec143ec0 con 0x7fe5ec143ae0 2023-11-24T06:08:47.101 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.095+0000 7fe5e959a700 1 -- 172.21.15.91:0/385684116 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d40035b0 con 0x7fe5ec143ae0 2023-11-24T06:08:47.101 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.095+0000 7fe5e959a700 1 -- 172.21.15.91:0/385684116 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299181409 0 0) 0x7fe5ec0b3ae0 con 0x7fe5ec0722c0 2023-11-24T06:08:47.101 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.095+0000 7fe5e959a700 1 -- 172.21.15.91:0/385684116 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5ec143ec0 con 0x7fe5ec0722c0 2023-11-24T06:08:47.101 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.095+0000 7fe5e959a700 1 -- 172.21.15.91:0/385684116 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1514141990 0 0) 0x7fe5d40035b0 con 0x7fe5ec143ae0 2023-11-24T06:08:47.101 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.095+0000 7fe5ead9d700 1 -- 172.21.15.91:0/385684116 >> v1:172.21.15.94:6789/0 conn(0x7fe5ec0722c0 legacy=0x7fe5ec06e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:47.101 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.095+0000 7fe5ead9d700 1 -- 172.21.15.91:0/385684116 >> v1:172.21.15.91:6789/0 conn(0x7fe5ec143ae0 legacy=0x7fe5ec071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:47.101 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.095+0000 7fe5ead9d700 1 -- 172.21.15.91:0/385684116 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec0b3dc0 con 0x7fe5ec1440f0 2023-11-24T06:08:47.101 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.095+0000 7fe5ead9d700 1 --2- 172.21.15.91:0/385684116 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5ec1440f0 0x7fe5ec0b51b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4009dc0 tx=0x7fe5e400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bd803f7767b2708 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:47.101 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.095+0000 7fe5e959a700 1 -- 172.21.15.91:0/385684116 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e400e070 con 0x7fe5ec1440f0 2023-11-24T06:08:47.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.095+0000 7fe5e959a700 1 -- 172.21.15.91:0/385684116 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5e4003a20 con 0x7fe5ec1440f0 2023-11-24T06:08:47.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.095+0000 7fe5e959a700 1 -- 172.21.15.91:0/385684116 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc001670 con 0x7fe5ec1440f0 2023-11-24T06:08:47.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.096+0000 7fe5f0abb700 1 -- 172.21.15.91:0/385684116 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5ec1440f0 msgr2=0x7fe5ec0b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:47.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.096+0000 7fe5f0abb700 1 --2- 172.21.15.91:0/385684116 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5ec1440f0 0x7fe5ec0b51b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe5e4009dc0 tx=0x7fe5e400b330 comp rx=0 tx=0).stop 2023-11-24T06:08:47.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.096+0000 7fe5f0abb700 1 -- 172.21.15.91:0/385684116 shutdown_connections 2023-11-24T06:08:47.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.096+0000 7fe5f0abb700 1 --2- 172.21.15.91:0/385684116 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5ec1440f0 0x7fe5ec0b51b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:47.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.096+0000 7fe5f0abb700 1 -- 172.21.15.91:0/385684116 >> 172.21.15.91:0/385684116 conn(0x7fe5ec0b2740 msgr2=0x7fe5ec0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:47.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.096+0000 7fe5f0abb700 1 -- 172.21.15.91:0/385684116 shutdown_connections 2023-11-24T06:08:47.102 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.096+0000 7fe5f0abb700 1 -- 172.21.15.91:0/385684116 wait complete. 2023-11-24T06:08:47.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.097+0000 7fe5f0abb700 1 Processor -- start 2023-11-24T06:08:47.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.097+0000 7fe5f0abb700 1 -- start start 2023-11-24T06:08:47.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.098+0000 7fe5f0abb700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe5ec0722c0 0x7fe5ec13ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:47.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.098+0000 7fe5f0abb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5ec143ae0 0x7fe5ec15d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:47.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.098+0000 7fe5f0abb700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5ec1440f0 0x7fe5ec157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:47.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.098+0000 7fe5f0abb700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec15c340 con 0x7fe5ec0722c0 2023-11-24T06:08:47.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.098+0000 7fe5f0abb700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec15c550 con 0x7fe5ec1440f0 2023-11-24T06:08:47.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.098+0000 7fe5f0abb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec15c830 con 0x7fe5ec143ae0 2023-11-24T06:08:47.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.098+0000 7fe5ea59c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe5ec0722c0 0x7fe5ec13ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:47.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.098+0000 7fe5ea59c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe5ec0722c0 0x7fe5ec13ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59330/0 (socket says 172.21.15.91:59330) 2023-11-24T06:08:47.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.098+0000 7fe5ea59c700 1 -- 172.21.15.91:0/2581392682 learned_addr learned my addr 172.21.15.91:0/2581392682 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:47.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.098+0000 7fe5ead9d700 1 --2- 172.21.15.91:0/2581392682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5ec1440f0 0x7fe5ec157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:47.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.099+0000 7fe5e9d9b700 1 --2- 172.21.15.91:0/2581392682 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5ec143ae0 0x7fe5ec15d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:47.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.099+0000 7fe5ea59c700 1 -- 172.21.15.91:0/2581392682 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5ec143ae0 msgr2=0x7fe5ec15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:47.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.099+0000 7fe5ea59c700 1 --2- 172.21.15.91:0/2581392682 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5ec143ae0 0x7fe5ec15d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:47.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.099+0000 7fe5ea59c700 1 -- 172.21.15.91:0/2581392682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5ec1440f0 msgr2=0x7fe5ec157030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:47.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.099+0000 7fe5ea59c700 1 --2- 172.21.15.91:0/2581392682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5ec1440f0 0x7fe5ec157030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:47.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.099+0000 7fe5ea59c700 1 -- 172.21.15.91:0/2581392682 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e4009a70 con 0x7fe5ec0722c0 2023-11-24T06:08:47.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.099+0000 7fe5ead9d700 1 --2- 172.21.15.91:0/2581392682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5ec1440f0 0x7fe5ec157030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:47.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.099+0000 7fe5e9d9b700 1 --2- 172.21.15.91:0/2581392682 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5ec143ae0 0x7fe5ec15d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:47.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.099+0000 7fe5ea59c700 1 --2- 172.21.15.91:0/2581392682 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe5ec0722c0 0x7fe5ec13ef00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc00c750 tx=0x7fe5dc00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:47.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.099+0000 7fe5daffd700 1 -- 172.21.15.91:0/2581392682 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc004c80 con 0x7fe5ec0722c0 2023-11-24T06:08:47.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.099+0000 7fe5daffd700 1 -- 172.21.15.91:0/2581392682 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5dc004de0 con 0x7fe5ec0722c0 2023-11-24T06:08:47.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.099+0000 7fe5daffd700 1 -- 172.21.15.91:0/2581392682 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5dc007310 con 0x7fe5ec0722c0 2023-11-24T06:08:47.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.099+0000 7fe5f0abb700 1 -- 172.21.15.91:0/2581392682 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5ec15cb10 con 0x7fe5ec0722c0 2023-11-24T06:08:47.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.099+0000 7fe5f0abb700 1 -- 172.21.15.91:0/2581392682 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5ec15d0d0 con 0x7fe5ec0722c0 2023-11-24T06:08:47.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.100+0000 7fe5f0abb700 1 -- 172.21.15.91:0/2581392682 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5ec075480 con 0x7fe5ec0722c0 2023-11-24T06:08:47.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.100+0000 7fe5daffd700 1 -- 172.21.15.91:0/2581392682 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe5dc007470 con 0x7fe5ec0722c0 2023-11-24T06:08:47.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.101+0000 7fe5daffd700 1 --2- 172.21.15.91:0/2581392682 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe5cc06aed0 0x7fe5cc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:47.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.101+0000 7fe5daffd700 1 -- 172.21.15.91:0/2581392682 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fe5dc0894c0 con 0x7fe5ec0722c0 2023-11-24T06:08:47.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.101+0000 7fe5e9d9b700 1 --2- 172.21.15.91:0/2581392682 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe5cc06aed0 0x7fe5cc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:47.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.101+0000 7fe5e9d9b700 1 --2- 172.21.15.91:0/2581392682 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe5cc06aed0 0x7fe5cc06d390 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0007fd0 tx=0x7fe5e0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:47.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.105+0000 7fe5daffd700 1 -- 172.21.15.91:0/2581392682 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5dc054530 con 0x7fe5ec0722c0 2023-11-24T06:08:47.344 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.340+0000 7fe5f0abb700 1 -- 172.21.15.91:0/2581392682 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe5ec15d440 con 0x7fe5ec0722c0 2023-11-24T06:08:47.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.959+0000 7fe5daffd700 1 -- 172.21.15.91:0/2581392682 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe5dc057b50 con 0x7fe5ec0722c0 2023-11-24T06:08:47.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.962+0000 7fe5f0abb700 1 -- 172.21.15.91:0/2581392682 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe5cc06aed0 msgr2=0x7fe5cc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:47.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.962+0000 7fe5f0abb700 1 --2- 172.21.15.91:0/2581392682 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe5cc06aed0 0x7fe5cc06d390 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0007fd0 tx=0x7fe5e0009040 comp rx=0 tx=0).stop 2023-11-24T06:08:47.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.962+0000 7fe5f0abb700 1 -- 172.21.15.91:0/2581392682 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe5ec0722c0 msgr2=0x7fe5ec13ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:47.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.962+0000 7fe5f0abb700 1 --2- 172.21.15.91:0/2581392682 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe5ec0722c0 0x7fe5ec13ef00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc00c750 tx=0x7fe5dc00cb10 comp rx=0 tx=0).stop 2023-11-24T06:08:47.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.962+0000 7fe5f0abb700 1 -- 172.21.15.91:0/2581392682 shutdown_connections 2023-11-24T06:08:47.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.962+0000 7fe5f0abb700 1 --2- 172.21.15.91:0/2581392682 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe5cc06aed0 0x7fe5cc06d390 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:47.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.962+0000 7fe5f0abb700 1 --2- 172.21.15.91:0/2581392682 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe5ec1440f0 0x7fe5ec157030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:47.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.962+0000 7fe5f0abb700 1 --2- 172.21.15.91:0/2581392682 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe5ec0722c0 0x7fe5ec13ef00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:47.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.962+0000 7fe5f0abb700 1 --2- 172.21.15.91:0/2581392682 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe5ec143ae0 0x7fe5ec15d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:47.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.962+0000 7fe5f0abb700 1 -- 172.21.15.91:0/2581392682 >> 172.21.15.91:0/2581392682 conn(0x7fe5ec0b2740 msgr2=0x7fe5ec06e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:47.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.962+0000 7fe5f0abb700 1 -- 172.21.15.91:0/2581392682 shutdown_connections 2023-11-24T06:08:47.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:47.962+0000 7fe5f0abb700 1 -- 172.21.15.91:0/2581392682 wait complete. 2023-11-24T06:08:47.980 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T06:08:48.148 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.145+0000 7fc12f914700 1 Processor -- start 2023-11-24T06:08:48.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.145+0000 7fc12f914700 1 -- start start 2023-11-24T06:08:48.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc12f914700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc12806afd0 0x7fc1280b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:48.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc12f914700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fc128072350 0x7fc1280727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:48.152 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc12f914700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc128071ec0 con 0x7fc12806b3b0 2023-11-24T06:08:48.152 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc12f914700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc128071b90 con 0x7fc128072350 2023-11-24T06:08:48.152 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc12f914700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12815ed30 con 0x7fc12806afd0 2023-11-24T06:08:48.152 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc12deb1700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fc128072350 0x7fc1280727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:48.152 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc12deb1700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fc128072350 0x7fc1280727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59340/0 (socket says 172.21.15.91:59340) 2023-11-24T06:08:48.153 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc12deb1700 1 -- 172.21.15.91:0/3373774641 learned_addr learned my addr 172.21.15.91:0/3373774641 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:48.153 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc12d6b0700 1 --2- 172.21.15.91:0/3373774641 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc12806afd0 0x7fc1280b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:48.153 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc12deb1700 1 -- 172.21.15.91:0/3373774641 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc12806afd0 msgr2=0x7fc1280b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:48.153 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc12deb1700 1 --2- 172.21.15.91:0/3373774641 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc12806afd0 0x7fc1280b59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:48.153 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc12deb1700 1 -- 172.21.15.91:0/3373774641 >> v1:172.21.15.94:6789/0 conn(0x7fc12806b3b0 legacy=0x7fc12815e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:48.153 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc12deb1700 1 -- 172.21.15.91:0/3373774641 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12815f010 con 0x7fc128072350 2023-11-24T06:08:48.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc12d6b0700 1 --2- 172.21.15.91:0/3373774641 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc12806afd0 0x7fc1280b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:48.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc12deb1700 1 --2- 172.21.15.91:0/3373774641 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc128072350 0x7fc1280727a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc120009e50 tx=0x7fc12000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bbfb661dfbf9eea server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:48.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc11ffff700 1 -- 172.21.15.91:0/3373774641 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12000e070 con 0x7fc128072350 2023-11-24T06:08:48.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc11ffff700 1 -- 172.21.15.91:0/3373774641 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc120003a20 con 0x7fc128072350 2023-11-24T06:08:48.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.146+0000 7fc11ffff700 1 -- 172.21.15.91:0/3373774641 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc120003dd0 con 0x7fc128072350 2023-11-24T06:08:48.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.147+0000 7fc12f914700 1 -- 172.21.15.91:0/3373774641 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc128072350 msgr2=0x7fc1280727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:48.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.147+0000 7fc12f914700 1 --2- 172.21.15.91:0/3373774641 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc128072350 0x7fc1280727a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc120009e50 tx=0x7fc12000b330 comp rx=0 tx=0).stop 2023-11-24T06:08:48.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.147+0000 7fc12f914700 1 -- 172.21.15.91:0/3373774641 shutdown_connections 2023-11-24T06:08:48.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.147+0000 7fc12f914700 1 --2- 172.21.15.91:0/3373774641 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc128072350 0x7fc1280727a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:48.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.147+0000 7fc12f914700 1 --2- 172.21.15.91:0/3373774641 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc12806afd0 0x7fc1280b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:48.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.147+0000 7fc12f914700 1 -- 172.21.15.91:0/3373774641 >> 172.21.15.91:0/3373774641 conn(0x7fc1280b2760 msgr2=0x7fc1280b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:48.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.147+0000 7fc12f914700 1 -- 172.21.15.91:0/3373774641 shutdown_connections 2023-11-24T06:08:48.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.147+0000 7fc12f914700 1 -- 172.21.15.91:0/3373774641 wait complete. 2023-11-24T06:08:48.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.148+0000 7fc12f914700 1 Processor -- start 2023-11-24T06:08:48.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.148+0000 7fc12f914700 1 -- start start 2023-11-24T06:08:48.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.149+0000 7fc12f914700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc12806afd0 0x7fc128131a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:48.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.149+0000 7fc12f914700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc12806b3b0 0x7fc1281f3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:48.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.149+0000 7fc12f914700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc128072350 0x7fc1281300c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:48.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.149+0000 7fc12f914700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1281309f0 con 0x7fc12806afd0 2023-11-24T06:08:48.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.149+0000 7fc12f914700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc128130cd0 con 0x7fc12806b3b0 2023-11-24T06:08:48.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.149+0000 7fc12f914700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc128130fb0 con 0x7fc128072350 2023-11-24T06:08:48.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.149+0000 7fc12d6b0700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc12806afd0 0x7fc128131a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:48.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.149+0000 7fc12d6b0700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc12806afd0 0x7fc128131a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59344/0 (socket says 172.21.15.91:59344) 2023-11-24T06:08:48.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.149+0000 7fc12d6b0700 1 -- 172.21.15.91:0/3885078397 learned_addr learned my addr 172.21.15.91:0/3885078397 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:48.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.149+0000 7fc12deb1700 1 --2- 172.21.15.91:0/3885078397 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc128072350 0x7fc1281300c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:48.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.149+0000 7fc12ceaf700 1 --2- 172.21.15.91:0/3885078397 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc12806b3b0 0x7fc1281f3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:48.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.149+0000 7fc12d6b0700 1 -- 172.21.15.91:0/3885078397 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc128072350 msgr2=0x7fc1281300c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:48.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.149+0000 7fc12d6b0700 1 --2- 172.21.15.91:0/3885078397 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc128072350 0x7fc1281300c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:48.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.149+0000 7fc12d6b0700 1 -- 172.21.15.91:0/3885078397 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc12806b3b0 msgr2=0x7fc1281f3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:48.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.149+0000 7fc12d6b0700 1 --2- 172.21.15.91:0/3885078397 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc12806b3b0 0x7fc1281f3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:48.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.149+0000 7fc12d6b0700 1 -- 172.21.15.91:0/3885078397 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc120009b30 con 0x7fc12806afd0 2023-11-24T06:08:48.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.150+0000 7fc12d6b0700 1 --2- 172.21.15.91:0/3885078397 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc12806afd0 0x7fc128131a90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc12400c750 tx=0x7fc12400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:48.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.150+0000 7fc11dffb700 1 -- 172.21.15.91:0/3885078397 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124011800 con 0x7fc12806afd0 2023-11-24T06:08:48.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.150+0000 7fc11dffb700 1 -- 172.21.15.91:0/3885078397 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc124011960 con 0x7fc12806afd0 2023-11-24T06:08:48.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.150+0000 7fc11dffb700 1 -- 172.21.15.91:0/3885078397 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124011cd0 con 0x7fc12806afd0 2023-11-24T06:08:48.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.150+0000 7fc12f914700 1 -- 172.21.15.91:0/3885078397 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc128131290 con 0x7fc12806afd0 2023-11-24T06:08:48.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.150+0000 7fc12f914700 1 -- 172.21.15.91:0/3885078397 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1281f3d90 con 0x7fc12806afd0 2023-11-24T06:08:48.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.151+0000 7fc12f914700 1 -- 172.21.15.91:0/3885078397 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1281318e0 con 0x7fc12806afd0 2023-11-24T06:08:48.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.151+0000 7fc11dffb700 1 -- 172.21.15.91:0/3885078397 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc124021350 con 0x7fc12806afd0 2023-11-24T06:08:48.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.152+0000 7fc11dffb700 1 --2- 172.21.15.91:0/3885078397 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc10c06aed0 0x7fc10c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:48.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.152+0000 7fc11dffb700 1 -- 172.21.15.91:0/3885078397 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fc124089450 con 0x7fc12806afd0 2023-11-24T06:08:48.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.152+0000 7fc12ceaf700 1 --2- 172.21.15.91:0/3885078397 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc10c06aed0 0x7fc10c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:48.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.152+0000 7fc12ceaf700 1 --2- 172.21.15.91:0/3885078397 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc10c06aed0 0x7fc10c06d390 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc128144170 tx=0x7fc118007ec0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:48.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.156+0000 7fc11dffb700 1 -- 172.21.15.91:0/3885078397 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1240544c0 con 0x7fc12806afd0 2023-11-24T06:08:48.351 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.347+0000 7fc12f914700 1 -- 172.21.15.91:0/3885078397 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc12806b810 con 0x7fc12806afd0 2023-11-24T06:08:48.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.963+0000 7fc11dffb700 1 -- 172.21.15.91:0/3885078397 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc12806b810 con 0x7fc12806afd0 2023-11-24T06:08:48.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.965+0000 7fc12f914700 1 -- 172.21.15.91:0/3885078397 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc10c06aed0 msgr2=0x7fc10c06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:48.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.965+0000 7fc12f914700 1 --2- 172.21.15.91:0/3885078397 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc10c06aed0 0x7fc10c06d390 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc128144170 tx=0x7fc118007ec0 comp rx=0 tx=0).stop 2023-11-24T06:08:48.970 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.965+0000 7fc12f914700 1 -- 172.21.15.91:0/3885078397 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc12806afd0 msgr2=0x7fc128131a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:48.970 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.965+0000 7fc12f914700 1 --2- 172.21.15.91:0/3885078397 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc12806afd0 0x7fc128131a90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc12400c750 tx=0x7fc12400ca60 comp rx=0 tx=0).stop 2023-11-24T06:08:48.970 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.965+0000 7fc12f914700 1 -- 172.21.15.91:0/3885078397 shutdown_connections 2023-11-24T06:08:48.970 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.965+0000 7fc12f914700 1 --2- 172.21.15.91:0/3885078397 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc10c06aed0 0x7fc10c06d390 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:48.970 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.965+0000 7fc12f914700 1 --2- 172.21.15.91:0/3885078397 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc128072350 0x7fc1281300c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:48.970 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.965+0000 7fc12f914700 1 --2- 172.21.15.91:0/3885078397 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc12806afd0 0x7fc128131a90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:48.970 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.965+0000 7fc12f914700 1 --2- 172.21.15.91:0/3885078397 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc12806b3b0 0x7fc1281f3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:48.970 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.965+0000 7fc12f914700 1 -- 172.21.15.91:0/3885078397 >> 172.21.15.91:0/3885078397 conn(0x7fc1280b2760 msgr2=0x7fc12806e180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:48.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.965+0000 7fc12f914700 1 -- 172.21.15.91:0/3885078397 shutdown_connections 2023-11-24T06:08:48.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:48.965+0000 7fc12f914700 1 -- 172.21.15.91:0/3885078397 wait complete. 2023-11-24T06:08:48.971 INFO:teuthology.orchestra.run.smithi091.stderr:pool 'cephfs_metadata' removed 2023-11-24T06:08:48.984 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T06:08:49.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.105+0000 7f2de755c700 1 Processor -- start 2023-11-24T06:08:49.110 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.105+0000 7f2de755c700 1 -- start start 2023-11-24T06:08:49.116 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.106+0000 7f2de755c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2de00b59d0 0x7f2de015e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:49.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.106+0000 7f2de755c700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de0071ec0 con 0x7f2de0072350 2023-11-24T06:08:49.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.106+0000 7f2de755c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de0071b90 con 0x7f2de006afd0 2023-11-24T06:08:49.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.106+0000 7f2de755c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de015ed30 con 0x7f2de00b59d0 2023-11-24T06:08:49.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.106+0000 7f2de5af9700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f2de0072350 0x7f2de006dbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54288/0 (socket says 172.21.15.91:54288) 2023-11-24T06:08:49.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.106+0000 7f2de5af9700 1 -- 172.21.15.91:0/1313813566 learned_addr learned my addr 172.21.15.91:0/1313813566 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:49.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.106+0000 7f2dd7fff700 1 -- 172.21.15.91:0/1313813566 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695050455 0 0) 0x7f2de0071ec0 con 0x7f2de0072350 2023-11-24T06:08:49.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.106+0000 7f2dd7fff700 1 -- 172.21.15.91:0/1313813566 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dcc0035b0 con 0x7f2de0072350 2023-11-24T06:08:49.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.106+0000 7f2de4af7700 1 --2- 172.21.15.91:0/1313813566 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2de00b59d0 0x7f2de015e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:49.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2dd7fff700 1 -- 172.21.15.91:0/1313813566 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3643862709 0 0) 0x7f2dcc0035b0 con 0x7f2de0072350 2023-11-24T06:08:49.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2dd7fff700 1 -- 172.21.15.91:0/1313813566 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2de00b59d0 msgr2=0x7f2de015e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:49.117 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2dd7fff700 1 --2- 172.21.15.91:0/1313813566 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2de00b59d0 0x7f2de015e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:49.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2dd7fff700 1 -- 172.21.15.91:0/1313813566 >> v1:172.21.15.94:6789/0 conn(0x7f2de006afd0 legacy=0x7f2de006b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:49.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2dd7fff700 1 -- 172.21.15.91:0/1313813566 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de015f010 con 0x7f2de0072350 2023-11-24T06:08:49.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2dd7fff700 1 -- 172.21.15.91:0/1313813566 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f2dd80019f0 con 0x7f2de0072350 2023-11-24T06:08:49.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2dd7fff700 1 -- 172.21.15.91:0/1313813566 >> v1:172.21.15.91:6789/0 conn(0x7f2de0072350 legacy=0x7f2de006dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:49.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2dd7fff700 1 --2- 172.21.15.91:0/1313813566 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2dcc003a30 0x7f2dcc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:49.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2dd7fff700 1 --2- 172.21.15.91:0/1313813566 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2de00b59d0 0x7f2dcc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:49.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2dd7fff700 1 --2- 172.21.15.91:0/1313813566 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2dcc004d90 0x7f2dcc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:49.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2dd7fff700 1 -- 172.21.15.91:0/1313813566 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de015ed30 con 0x7f2dcc003a30 2023-11-24T06:08:49.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2dd7fff700 1 -- 172.21.15.91:0/1313813566 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc00bac0 con 0x7f2dcc004d90 2023-11-24T06:08:49.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2dd7fff700 1 -- 172.21.15.91:0/1313813566 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc00bc80 con 0x7f2de00b59d0 2023-11-24T06:08:49.118 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2de52f8700 1 --2- 172.21.15.91:0/1313813566 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2dcc003a30 0x7f2dcc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:49.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2de5af9700 1 --2- 172.21.15.91:0/1313813566 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2dcc004d90 0x7f2dcc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:49.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2de4af7700 1 --2- 172.21.15.91:0/1313813566 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2de00b59d0 0x7f2dcc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:49.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2de52f8700 1 -- 172.21.15.91:0/1313813566 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2de00b59d0 msgr2=0x7f2dcc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:49.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2de52f8700 1 --2- 172.21.15.91:0/1313813566 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2de00b59d0 0x7f2dcc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:49.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2de52f8700 1 -- 172.21.15.91:0/1313813566 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2dcc004d90 msgr2=0x7f2dcc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:49.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2de52f8700 1 --2- 172.21.15.91:0/1313813566 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2dcc004d90 0x7f2dcc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:49.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2de52f8700 1 -- 172.21.15.91:0/1313813566 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc0035b0 con 0x7f2dcc003a30 2023-11-24T06:08:49.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2de52f8700 1 --2- 172.21.15.91:0/1313813566 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2dcc003a30 0x7f2dcc003e50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0007fd0 tx=0x7f2dcc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e344cf2667e80e3d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:49.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.107+0000 7f2dd7fff700 1 -- 172.21.15.91:0/1313813566 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd00052d0 con 0x7f2dcc003a30 2023-11-24T06:08:49.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.108+0000 7f2dd7fff700 1 -- 172.21.15.91:0/1313813566 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd0005470 con 0x7f2dcc003a30 2023-11-24T06:08:49.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.108+0000 7f2dd7fff700 1 -- 172.21.15.91:0/1313813566 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd00038c0 con 0x7f2dcc003a30 2023-11-24T06:08:49.119 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.108+0000 7f2de755c700 1 -- 172.21.15.91:0/1313813566 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2dcc003a30 msgr2=0x7f2dcc003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:49.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.108+0000 7f2de755c700 1 --2- 172.21.15.91:0/1313813566 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2dcc003a30 0x7f2dcc003e50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0007fd0 tx=0x7f2dcc005880 comp rx=0 tx=0).stop 2023-11-24T06:08:49.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.108+0000 7f2de5af9700 1 -- 172.21.15.91:0/1313813566 reap_dead start 2023-11-24T06:08:49.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.108+0000 7f2de755c700 1 -- 172.21.15.91:0/1313813566 shutdown_connections 2023-11-24T06:08:49.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.108+0000 7f2de755c700 1 -- 172.21.15.91:0/1313813566 >> 172.21.15.91:0/1313813566 conn(0x7f2de00b2760 msgr2=0x7f2de00b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:49.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.108+0000 7f2de755c700 1 -- 172.21.15.91:0/1313813566 shutdown_connections 2023-11-24T06:08:49.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.108+0000 7f2de755c700 1 -- 172.21.15.91:0/1313813566 wait complete. 2023-11-24T06:08:49.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.109+0000 7f2de755c700 1 Processor -- start 2023-11-24T06:08:49.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.109+0000 7f2de755c700 1 -- start start 2023-11-24T06:08:49.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.110+0000 7f2de755c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2dcc003a30 0x7f2de01426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:49.120 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.110+0000 7f2de755c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2de00ae830 0x7f2de00b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:49.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.110+0000 7f2de755c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2de00b1f70 0x7f2de013d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:49.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.110+0000 7f2de755c700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de00ad860 con 0x7f2dcc003a30 2023-11-24T06:08:49.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.110+0000 7f2de755c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de015d8e0 con 0x7f2de00b1f70 2023-11-24T06:08:49.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.110+0000 7f2de755c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de00ada70 con 0x7f2de00ae830 2023-11-24T06:08:49.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.110+0000 7f2de52f8700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2dcc003a30 0x7f2de01426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:49.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.110+0000 7f2de52f8700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2dcc003a30 0x7f2de01426b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59362/0 (socket says 172.21.15.91:59362) 2023-11-24T06:08:49.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.110+0000 7f2de52f8700 1 -- 172.21.15.91:0/2332232161 learned_addr learned my addr 172.21.15.91:0/2332232161 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:49.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.110+0000 7f2de4af7700 1 --2- 172.21.15.91:0/2332232161 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2de00ae830 0x7f2de00b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:49.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.110+0000 7f2de5af9700 1 --2- 172.21.15.91:0/2332232161 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2de00b1f70 0x7f2de013d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:49.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.110+0000 7f2de52f8700 1 -- 172.21.15.91:0/2332232161 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2de00ae830 msgr2=0x7f2de00b1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:49.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.110+0000 7f2de52f8700 1 --2- 172.21.15.91:0/2332232161 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2de00ae830 0x7f2de00b1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:49.121 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.110+0000 7f2de52f8700 1 -- 172.21.15.91:0/2332232161 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2de00b1f70 msgr2=0x7f2de013d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:49.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.110+0000 7f2de52f8700 1 --2- 172.21.15.91:0/2332232161 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2de00b1f70 0x7f2de013d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:49.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.110+0000 7f2de52f8700 1 -- 172.21.15.91:0/2332232161 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de00add50 con 0x7f2dcc003a30 2023-11-24T06:08:49.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.111+0000 7f2de52f8700 1 --2- 172.21.15.91:0/2332232161 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2dcc003a30 0x7f2de01426b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0004000 tx=0x7f2dd0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:49.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.111+0000 7f2dd5ffb700 1 -- 172.21.15.91:0/2332232161 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0007240 con 0x7f2dcc003a30 2023-11-24T06:08:49.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.111+0000 7f2dd5ffb700 1 -- 172.21.15.91:0/2332232161 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd00073a0 con 0x7f2dcc003a30 2023-11-24T06:08:49.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.111+0000 7f2de755c700 1 -- 172.21.15.91:0/2332232161 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de00ae030 con 0x7f2dcc003a30 2023-11-24T06:08:49.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.111+0000 7f2dd5ffb700 1 -- 172.21.15.91:0/2332232161 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0001290 con 0x7f2dcc003a30 2023-11-24T06:08:49.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.111+0000 7f2de755c700 1 -- 172.21.15.91:0/2332232161 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de0142bf0 con 0x7f2dcc003a30 2023-11-24T06:08:49.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.112+0000 7f2de755c700 1 -- 172.21.15.91:0/2332232161 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de00b25a0 con 0x7f2dcc003a30 2023-11-24T06:08:49.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.112+0000 7f2dd5ffb700 1 -- 172.21.15.91:0/2332232161 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2dd0003680 con 0x7f2dcc003a30 2023-11-24T06:08:49.122 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.113+0000 7f2dd5ffb700 1 --2- 172.21.15.91:0/2332232161 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2dc406aed0 0x7f2dc406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:49.123 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.113+0000 7f2dd5ffb700 1 -- 172.21.15.91:0/2332232161 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5473+0+0 (secure 0 0 0) 0x7f2dd0089ff0 con 0x7f2dcc003a30 2023-11-24T06:08:49.123 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.113+0000 7f2de4af7700 1 --2- 172.21.15.91:0/2332232161 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2dc406aed0 0x7f2dc406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:49.123 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.113+0000 7f2de4af7700 1 --2- 172.21.15.91:0/2332232161 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2dc406aed0 0x7f2dc406d390 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2de00b12f0 tx=0x7f2ddc009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:49.123 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.117+0000 7f2dd5ffb700 1 -- 172.21.15.91:0/2332232161 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2dd0055330 con 0x7f2dcc003a30 2023-11-24T06:08:49.312 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.309+0000 7f2de755c700 1 -- 172.21.15.91:0/2332232161 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2de006e260 con 0x7f2dcc003a30 2023-11-24T06:08:49.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.967+0000 7f2dd5ffb700 1 -- 172.21.15.91:0/2332232161 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2dd0058950 con 0x7f2dcc003a30 2023-11-24T06:08:49.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.970+0000 7f2de755c700 1 -- 172.21.15.91:0/2332232161 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2dc406aed0 msgr2=0x7f2dc406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:49.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.970+0000 7f2de755c700 1 --2- 172.21.15.91:0/2332232161 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2dc406aed0 0x7f2dc406d390 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2de00b12f0 tx=0x7f2ddc009210 comp rx=0 tx=0).stop 2023-11-24T06:08:49.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.970+0000 7f2de755c700 1 -- 172.21.15.91:0/2332232161 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2dcc003a30 msgr2=0x7f2de01426b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:49.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.970+0000 7f2de755c700 1 --2- 172.21.15.91:0/2332232161 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2dcc003a30 0x7f2de01426b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2dd0004000 tx=0x7f2dd0001120 comp rx=0 tx=0).stop 2023-11-24T06:08:49.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.970+0000 7f2de755c700 1 -- 172.21.15.91:0/2332232161 shutdown_connections 2023-11-24T06:08:49.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.970+0000 7f2de755c700 1 --2- 172.21.15.91:0/2332232161 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2dc406aed0 0x7f2dc406d390 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:49.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.970+0000 7f2de755c700 1 --2- 172.21.15.91:0/2332232161 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2de00b1f70 0x7f2de013d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:49.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.970+0000 7f2de755c700 1 --2- 172.21.15.91:0/2332232161 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2dcc003a30 0x7f2de01426b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:49.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.970+0000 7f2de755c700 1 --2- 172.21.15.91:0/2332232161 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2de00ae830 0x7f2de00b1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:49.975 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.970+0000 7f2de755c700 1 -- 172.21.15.91:0/2332232161 >> 172.21.15.91:0/2332232161 conn(0x7f2de00b2760 msgr2=0x7f2de006e620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:49.975 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.970+0000 7f2de755c700 1 -- 172.21.15.91:0/2332232161 shutdown_connections 2023-11-24T06:08:49.975 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:49.970+0000 7f2de755c700 1 -- 172.21.15.91:0/2332232161 wait complete. 2023-11-24T06:08:49.975 INFO:teuthology.orchestra.run.smithi091.stderr:pool 'cephfs_data' removed 2023-11-24T06:08:49.989 INFO:tasks.ceph.mds.a:Restarting daemon 2023-11-24T06:08:49.989 INFO:tasks.ceph.mds.a:Stopping old one... 2023-11-24T06:08:49.990 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T06:08:49.990 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:08:49.990 INFO:tasks.ceph.mds.c:Restarting daemon 2023-11-24T06:08:49.990 INFO:tasks.ceph.mds.c:Stopping old one... 2023-11-24T06:08:49.990 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T06:08:49.990 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:08:49.990 INFO:tasks.ceph.mds.b:Restarting daemon 2023-11-24T06:08:49.990 INFO:tasks.ceph.mds.b:Stopping old one... 2023-11-24T06:08:49.991 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T06:08:49.991 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:08:49.991 INFO:tasks.ceph.mds.d:Restarting daemon 2023-11-24T06:08:49.991 INFO:tasks.ceph.mds.d:Stopping old one... 2023-11-24T06:08:49.991 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T06:08:49.991 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:08:50.001 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T06:08:50.002 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-11-24T06:08:50.002 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T06:08:50.002 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-11-24T06:08:50.003 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T06:08:50.003 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-11-24T06:08:50.004 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T06:08:50.004 DEBUG:teuthology.orchestra.run.smithi094:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-11-24T06:08:50.044 INFO:tasks.ceph.mds.c:Started 2023-11-24T06:08:50.044 DEBUG:teuthology.parallel:result is None 2023-11-24T06:08:50.045 INFO:tasks.ceph.mds.b:Started 2023-11-24T06:08:50.045 INFO:tasks.ceph.mds.a:Started 2023-11-24T06:08:50.046 INFO:tasks.ceph.mds.d:Started 2023-11-24T06:08:50.046 DEBUG:teuthology.parallel:result is None 2023-11-24T06:08:50.046 DEBUG:teuthology.parallel:result is None 2023-11-24T06:08:50.046 DEBUG:teuthology.parallel:result is None 2023-11-24T06:08:50.046 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-11-24T06:08:50.123 INFO:tasks.ceph.mds.c.smithi091.stdout:starting mds.c at 2023-11-24T06:08:50.124 INFO:tasks.ceph.mds.a.smithi091.stdout:starting mds.a at 2023-11-24T06:08:50.130 INFO:tasks.ceph.mds.b.smithi094.stdout:starting mds.b at 2023-11-24T06:08:50.131 INFO:tasks.ceph.mds.d.smithi094.stdout:starting mds.d at 2023-11-24T06:08:50.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.169+0000 7f79794bf700 1 Processor -- start 2023-11-24T06:08:50.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.169+0000 7f79794bf700 1 -- start start 2023-11-24T06:08:50.175 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f79794bf700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f797415e460 0x7f79740d49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:50.175 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f79794bf700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f79740cf480 0x7f79740d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:50.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f79794bf700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79740d29d0 0x7f79740d0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:50.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f79794bf700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79740d2e60 con 0x7f797415e460 2023-11-24T06:08:50.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f79794bf700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79740b3370 con 0x7f79740cf480 2023-11-24T06:08:50.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f79794bf700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79740cfc40 con 0x7f79740d29d0 2023-11-24T06:08:50.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f7972ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f797415e460 0x7f79740d49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:50.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f7972ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f797415e460 0x7f79740d49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59392/0 (socket says 172.21.15.91:59392) 2023-11-24T06:08:50.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f7972ffd700 1 -- 172.21.15.91:0/3288023973 learned_addr learned my addr 172.21.15.91:0/3288023973 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:50.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f7972ffd700 1 -- 172.21.15.91:0/3288023973 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79740d29d0 msgr2=0x7f79740d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:50.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f79737fe700 1 --2- 172.21.15.91:0/3288023973 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79740d29d0 0x7f79740d0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:50.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f79727fc700 1 --2- 172.21.15.91:0/3288023973 >> v2:172.21.15.94:3300/0 conn(0x7f79740cf480 0x7f79740d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:50.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f7972ffd700 1 --2- 172.21.15.91:0/3288023973 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79740d29d0 0x7f79740d0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:50.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f7972ffd700 1 -- 172.21.15.91:0/3288023973 >> v2:172.21.15.94:3300/0 conn(0x7f79740cf480 msgr2=0x7f79740d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:50.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f7972ffd700 1 --2- 172.21.15.91:0/3288023973 >> v2:172.21.15.94:3300/0 conn(0x7f79740cf480 0x7f79740d2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:50.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f7972ffd700 1 -- 172.21.15.91:0/3288023973 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79740b35c0 con 0x7f797415e460 2023-11-24T06:08:50.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f79737fe700 1 --2- 172.21.15.91:0/3288023973 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79740d29d0 0x7f79740d0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:50.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f79727fc700 1 --2- 172.21.15.91:0/3288023973 >> v2:172.21.15.94:3300/0 conn(0x7f79740cf480 0x7f79740d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:50.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.170+0000 7f7972ffd700 1 --2- 172.21.15.91:0/3288023973 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f797415e460 0x7f79740d49d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f797408e2f0 tx=0x7f7964009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c29612930e43a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:50.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.171+0000 7f7971ffb700 1 -- 172.21.15.91:0/3288023973 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796400b870 con 0x7f797415e460 2023-11-24T06:08:50.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.171+0000 7f7971ffb700 1 -- 172.21.15.91:0/3288023973 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796400b9d0 con 0x7f797415e460 2023-11-24T06:08:50.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.171+0000 7f7971ffb700 1 -- 172.21.15.91:0/3288023973 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796400bd60 con 0x7f797415e460 2023-11-24T06:08:50.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.171+0000 7f79794bf700 1 -- 172.21.15.91:0/3288023973 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f797415e460 msgr2=0x7f79740d49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:50.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.171+0000 7f79794bf700 1 --2- 172.21.15.91:0/3288023973 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f797415e460 0x7f79740d49d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f797408e2f0 tx=0x7f7964009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:50.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.171+0000 7f79794bf700 1 -- 172.21.15.91:0/3288023973 shutdown_connections 2023-11-24T06:08:50.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.171+0000 7f79794bf700 1 --2- 172.21.15.91:0/3288023973 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79740d29d0 0x7f79740d0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:50.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.171+0000 7f79794bf700 1 --2- 172.21.15.91:0/3288023973 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f797415e460 0x7f79740d49d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:50.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.171+0000 7f79794bf700 1 --2- 172.21.15.91:0/3288023973 >> v2:172.21.15.94:3300/0 conn(0x7f79740cf480 0x7f79740d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:50.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.171+0000 7f79794bf700 1 -- 172.21.15.91:0/3288023973 >> 172.21.15.91:0/3288023973 conn(0x7f79740d3eb0 msgr2=0x7f79740d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:50.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.171+0000 7f79794bf700 1 -- 172.21.15.91:0/3288023973 shutdown_connections 2023-11-24T06:08:50.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.171+0000 7f79794bf700 1 -- 172.21.15.91:0/3288023973 wait complete. 2023-11-24T06:08:50.180 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.173+0000 7f79794bf700 1 Processor -- start 2023-11-24T06:08:50.181 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.173+0000 7f79794bf700 1 -- start start 2023-11-24T06:08:50.181 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.173+0000 7f79794bf700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f79740cf480 0x7f797415d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:50.181 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.173+0000 7f79794bf700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79740d29d0 0x7f797415c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:50.181 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.173+0000 7f79794bf700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f797415e460 0x7f79740c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:50.181 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.173+0000 7f79794bf700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79740c8500 con 0x7f79740cf480 2023-11-24T06:08:50.181 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.173+0000 7f79794bf700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79740c7b60 con 0x7f797415e460 2023-11-24T06:08:50.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.173+0000 7f79794bf700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79740c7e40 con 0x7f79740d29d0 2023-11-24T06:08:50.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.173+0000 7f7972ffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f79740cf480 0x7f797415d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:50.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.173+0000 7f7972ffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f79740cf480 0x7f797415d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59398/0 (socket says 172.21.15.91:59398) 2023-11-24T06:08:50.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.173+0000 7f7972ffd700 1 -- 172.21.15.91:0/364166842 learned_addr learned my addr 172.21.15.91:0/364166842 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:50.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.174+0000 7f79737fe700 1 --2- 172.21.15.91:0/364166842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f797415e460 0x7f79740c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:50.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.174+0000 7f79727fc700 1 --2- 172.21.15.91:0/364166842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79740d29d0 0x7f797415c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:50.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.174+0000 7f7972ffd700 1 -- 172.21.15.91:0/364166842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79740d29d0 msgr2=0x7f797415c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:50.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.174+0000 7f7972ffd700 1 --2- 172.21.15.91:0/364166842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79740d29d0 0x7f797415c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:50.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.174+0000 7f7972ffd700 1 -- 172.21.15.91:0/364166842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f797415e460 msgr2=0x7f79740c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:50.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.174+0000 7f7972ffd700 1 --2- 172.21.15.91:0/364166842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f797415e460 0x7f79740c8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:50.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.174+0000 7f7972ffd700 1 -- 172.21.15.91:0/364166842 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7964009550 con 0x7f79740cf480 2023-11-24T06:08:50.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.174+0000 7f79727fc700 1 --2- 172.21.15.91:0/364166842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79740d29d0 0x7f797415c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:50.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.174+0000 7f7972ffd700 1 --2- 172.21.15.91:0/364166842 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f79740cf480 0x7f797415d000 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7964014040 tx=0x7f7964004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:50.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.174+0000 7f795b7fe700 1 -- 172.21.15.91:0/364166842 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7964004310 con 0x7f79740cf480 2023-11-24T06:08:50.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.174+0000 7f795b7fe700 1 -- 172.21.15.91:0/364166842 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796401e070 con 0x7f79740cf480 2023-11-24T06:08:50.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.174+0000 7f795b7fe700 1 -- 172.21.15.91:0/364166842 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79640036a0 con 0x7f79740cf480 2023-11-24T06:08:50.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.174+0000 7f79794bf700 1 -- 172.21.15.91:0/364166842 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79740c8120 con 0x7f79740cf480 2023-11-24T06:08:50.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.174+0000 7f79794bf700 1 -- 172.21.15.91:0/364166842 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f797415cc80 con 0x7f79740cf480 2023-11-24T06:08:50.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.175+0000 7f79794bf700 1 -- 172.21.15.91:0/364166842 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7974067f60 con 0x7f79740cf480 2023-11-24T06:08:50.185 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.176+0000 7f795b7fe700 1 -- 172.21.15.91:0/364166842 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7964020030 con 0x7f79740cf480 2023-11-24T06:08:50.185 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.176+0000 7f795b7fe700 1 --2- 172.21.15.91:0/364166842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f796006aed0 0x7f796006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:50.185 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.176+0000 7f795b7fe700 1 -- 172.21.15.91:0/364166842 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f796408a2f0 con 0x7f79740cf480 2023-11-24T06:08:50.185 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.176+0000 7f79727fc700 1 --2- 172.21.15.91:0/364166842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f796006aed0 0x7f796006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:50.185 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.176+0000 7f79727fc700 1 --2- 172.21.15.91:0/364166842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f796006aed0 0x7f796006d390 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f79740b30e0 tx=0x7f7968008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:50.186 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.180+0000 7f795b7fe700 1 -- 172.21.15.91:0/364166842 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7964055730 con 0x7f79740cf480 2023-11-24T06:08:50.392 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:50.388+0000 7f79794bf700 1 -- 172.21.15.91:0/364166842 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f79740d2270 con 0x7f79740cf480 2023-11-24T06:08:51.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.739+0000 7f795b7fe700 1 -- 172.21.15.91:0/364166842 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v30) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7964058d50 con 0x7f79740cf480 2023-11-24T06:08:51.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.742+0000 7f79794bf700 1 -- 172.21.15.91:0/364166842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f796006aed0 msgr2=0x7f796006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:51.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.742+0000 7f79794bf700 1 --2- 172.21.15.91:0/364166842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f796006aed0 0x7f796006d390 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f79740b30e0 tx=0x7f7968008040 comp rx=0 tx=0).stop 2023-11-24T06:08:51.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.742+0000 7f79794bf700 1 -- 172.21.15.91:0/364166842 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f79740cf480 msgr2=0x7f797415d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:51.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.742+0000 7f79794bf700 1 --2- 172.21.15.91:0/364166842 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f79740cf480 0x7f797415d000 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7964014040 tx=0x7f7964004750 comp rx=0 tx=0).stop 2023-11-24T06:08:51.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.742+0000 7f79794bf700 1 -- 172.21.15.91:0/364166842 shutdown_connections 2023-11-24T06:08:51.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.742+0000 7f79794bf700 1 --2- 172.21.15.91:0/364166842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f796006aed0 0x7f796006d390 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:51.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.742+0000 7f79794bf700 1 --2- 172.21.15.91:0/364166842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f797415e460 0x7f79740c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:51.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.742+0000 7f79794bf700 1 --2- 172.21.15.91:0/364166842 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f79740cf480 0x7f797415d000 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:51.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.742+0000 7f79794bf700 1 --2- 172.21.15.91:0/364166842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79740d29d0 0x7f797415c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:51.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.743+0000 7f79794bf700 1 -- 172.21.15.91:0/364166842 >> 172.21.15.91:0/364166842 conn(0x7f79740d3eb0 msgr2=0x7f79740ceb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:51.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.743+0000 7f79794bf700 1 -- 172.21.15.91:0/364166842 shutdown_connections 2023-11-24T06:08:51.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.743+0000 7f79794bf700 1 -- 172.21.15.91:0/364166842 wait complete. 2023-11-24T06:08:51.748 INFO:teuthology.orchestra.run.smithi091.stderr: removed all blocklist entries 2023-11-24T06:08:51.761 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-11-24T06:08:51.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.883+0000 7f93cfbe6700 1 Processor -- start 2023-11-24T06:08:51.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.883+0000 7f93cfbe6700 1 -- start start 2023-11-24T06:08:51.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cfbe6700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f93c815e460 0x7f93c80d49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:51.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cfbe6700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c80b3370 con 0x7f93c80d2ba0 2023-11-24T06:08:51.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cfbe6700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c80cfc40 con 0x7f93c80cf480 2023-11-24T06:08:51.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cfbe6700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c80b35c0 con 0x7f93c815e460 2023-11-24T06:08:51.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93ce183700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f93c80d2ba0 0x7f93c80b40a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54348/0 (socket says 172.21.15.91:54348) 2023-11-24T06:08:51.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93ce183700 1 -- 172.21.15.91:0/2649710900 learned_addr learned my addr 172.21.15.91:0/2649710900 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:51.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cd982700 1 --2- 172.21.15.91:0/2649710900 >> v2:172.21.15.94:3300/0 conn(0x7f93c815e460 0x7f93c80d49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:51.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cc980700 1 -- 172.21.15.91:0/2649710900 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669547763 0 0) 0x7f93c80b3370 con 0x7f93c80d2ba0 2023-11-24T06:08:51.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cc980700 1 -- 172.21.15.91:0/2649710900 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93b40035b0 con 0x7f93c80d2ba0 2023-11-24T06:08:51.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cc980700 1 -- 172.21.15.91:0/2649710900 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2972877879 0 0) 0x7f93b40035b0 con 0x7f93c80d2ba0 2023-11-24T06:08:51.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cc980700 1 -- 172.21.15.91:0/2649710900 >> v2:172.21.15.94:3300/0 conn(0x7f93c815e460 msgr2=0x7f93c80d49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:51.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cc980700 1 --2- 172.21.15.91:0/2649710900 >> v2:172.21.15.94:3300/0 conn(0x7f93c815e460 0x7f93c80d49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:51.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cc980700 1 -- 172.21.15.91:0/2649710900 >> v1:172.21.15.94:6789/0 conn(0x7f93c80cf480 legacy=0x7f93c80d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:51.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cc980700 1 -- 172.21.15.91:0/2649710900 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c80b38a0 con 0x7f93c80d2ba0 2023-11-24T06:08:51.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cc980700 1 -- 172.21.15.91:0/2649710900 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f93c00019f0 con 0x7f93c80d2ba0 2023-11-24T06:08:51.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cc980700 1 -- 172.21.15.91:0/2649710900 >> v1:172.21.15.91:6789/0 conn(0x7f93c80d2ba0 legacy=0x7f93c80b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:51.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cc980700 1 --2- 172.21.15.91:0/2649710900 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f93b4003a00 0x7f93b4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:51.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cc980700 1 --2- 172.21.15.91:0/2649710900 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93b4004850 0x7f93b4004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:51.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cc980700 1 --2- 172.21.15.91:0/2649710900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93b4005210 0x7f93b400ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:51.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cc980700 1 -- 172.21.15.91:0/2649710900 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c80b35c0 con 0x7f93b4004850 2023-11-24T06:08:51.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cc980700 1 -- 172.21.15.91:0/2649710900 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b400c0c0 con 0x7f93b4005210 2023-11-24T06:08:51.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.884+0000 7f93cc980700 1 -- 172.21.15.91:0/2649710900 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b400c280 con 0x7f93b4003a00 2023-11-24T06:08:51.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.885+0000 7f93cd181700 1 --2- 172.21.15.91:0/2649710900 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93b4004850 0x7f93b4004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:51.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.885+0000 7f93cd982700 1 --2- 172.21.15.91:0/2649710900 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f93b4003a00 0x7f93b4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:51.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.885+0000 7f93ce183700 1 --2- 172.21.15.91:0/2649710900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93b4005210 0x7f93b400ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:51.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.885+0000 7f93cd181700 1 -- 172.21.15.91:0/2649710900 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f93b4003a00 msgr2=0x7f93b4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:51.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.885+0000 7f93cd181700 1 --2- 172.21.15.91:0/2649710900 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f93b4003a00 0x7f93b4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:51.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.885+0000 7f93cd181700 1 -- 172.21.15.91:0/2649710900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93b4005210 msgr2=0x7f93b400ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:51.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.885+0000 7f93cd181700 1 --2- 172.21.15.91:0/2649710900 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93b4005210 0x7f93b400ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:51.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.885+0000 7f93cd181700 1 -- 172.21.15.91:0/2649710900 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b40035b0 con 0x7f93b4004850 2023-11-24T06:08:51.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.885+0000 7f93ce183700 1 -- 172.21.15.91:0/2649710900 reap_dead start 2023-11-24T06:08:51.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.885+0000 7f93cd181700 1 --2- 172.21.15.91:0/2649710900 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93b4004850 0x7f93b4004cd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f93b80017c0 tx=0x7f93b4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfcbbd4552c2ac3a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:51.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.885+0000 7f93cc980700 1 -- 172.21.15.91:0/2649710900 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b80052d0 con 0x7f93b4004850 2023-11-24T06:08:51.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.885+0000 7f93cc980700 1 -- 172.21.15.91:0/2649710900 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b8005450 con 0x7f93b4004850 2023-11-24T06:08:51.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.885+0000 7f93cc980700 1 -- 172.21.15.91:0/2649710900 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b80038c0 con 0x7f93b4004850 2023-11-24T06:08:51.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.886+0000 7f93cfbe6700 1 -- 172.21.15.91:0/2649710900 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93b4004850 msgr2=0x7f93b4004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:51.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.886+0000 7f93cfbe6700 1 --2- 172.21.15.91:0/2649710900 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93b4004850 0x7f93b4004cd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f93b80017c0 tx=0x7f93b4005880 comp rx=0 tx=0).stop 2023-11-24T06:08:51.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.886+0000 7f93cfbe6700 1 -- 172.21.15.91:0/2649710900 shutdown_connections 2023-11-24T06:08:51.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.886+0000 7f93cfbe6700 1 --2- 172.21.15.91:0/2649710900 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93b4004850 0x7f93b4004cd0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:51.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.886+0000 7f93cfbe6700 1 -- 172.21.15.91:0/2649710900 >> 172.21.15.91:0/2649710900 conn(0x7f93c80d3eb0 msgr2=0x7f93c80d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:51.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.886+0000 7f93cfbe6700 1 -- 172.21.15.91:0/2649710900 shutdown_connections 2023-11-24T06:08:51.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.886+0000 7f93cfbe6700 1 -- 172.21.15.91:0/2649710900 wait complete. 2023-11-24T06:08:51.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.887+0000 7f93cfbe6700 1 Processor -- start 2023-11-24T06:08:51.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.887+0000 7f93cfbe6700 1 -- start start 2023-11-24T06:08:51.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cfbe6700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93b4004850 0x7f93c81525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:51.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cfbe6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f93c80cb310 0x7f93c81600c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:51.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cfbe6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93c80cb790 0x7f93c8160600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:51.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cfbe6700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c81531e0 con 0x7f93b4004850 2023-11-24T06:08:51.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cfbe6700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c80b6c30 con 0x7f93c80cb790 2023-11-24T06:08:51.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cfbe6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c80b6f10 con 0x7f93c80cb310 2023-11-24T06:08:51.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cd982700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93b4004850 0x7f93c81525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:51.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cd982700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93b4004850 0x7f93c81525e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59422/0 (socket says 172.21.15.91:59422) 2023-11-24T06:08:51.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cd982700 1 -- 172.21.15.91:0/1588530076 learned_addr learned my addr 172.21.15.91:0/1588530076 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:51.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cd181700 1 --2- 172.21.15.91:0/1588530076 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f93c80cb310 0x7f93c81600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:51.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93ce183700 1 --2- 172.21.15.91:0/1588530076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93c80cb790 0x7f93c8160600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:51.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cd982700 1 -- 172.21.15.91:0/1588530076 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f93c80cb310 msgr2=0x7f93c81600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:51.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cd982700 1 --2- 172.21.15.91:0/1588530076 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f93c80cb310 0x7f93c81600c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:51.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cd982700 1 -- 172.21.15.91:0/1588530076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93c80cb790 msgr2=0x7f93c8160600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:51.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cd982700 1 --2- 172.21.15.91:0/1588530076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93c80cb790 0x7f93c8160600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:51.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cd982700 1 -- 172.21.15.91:0/1588530076 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c80b71f0 con 0x7f93b4004850 2023-11-24T06:08:51.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cd982700 1 --2- 172.21.15.91:0/1588530076 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93b4004850 0x7f93c81525e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f93c40119a0 tx=0x7f93c4011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:51.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93be7fc700 1 -- 172.21.15.91:0/1588530076 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c400f940 con 0x7f93b4004850 2023-11-24T06:08:51.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93be7fc700 1 -- 172.21.15.91:0/1588530076 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c400faa0 con 0x7f93b4004850 2023-11-24T06:08:51.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cfbe6700 1 -- 172.21.15.91:0/1588530076 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c80b74d0 con 0x7f93b4004850 2023-11-24T06:08:51.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93be7fc700 1 -- 172.21.15.91:0/1588530076 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c400fe30 con 0x7f93b4004850 2023-11-24T06:08:51.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.888+0000 7f93cfbe6700 1 -- 172.21.15.91:0/1588530076 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c815f250 con 0x7f93b4004850 2023-11-24T06:08:51.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.889+0000 7f93cfbe6700 1 -- 172.21.15.91:0/1588530076 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c8067f60 con 0x7f93b4004850 2023-11-24T06:08:51.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.890+0000 7f93be7fc700 1 -- 172.21.15.91:0/1588530076 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f93c4014440 con 0x7f93b4004850 2023-11-24T06:08:51.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.890+0000 7f93be7fc700 1 --2- 172.21.15.91:0/1588530076 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93ac06aed0 0x7f93ac06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:51.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.890+0000 7f93be7fc700 1 -- 172.21.15.91:0/1588530076 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f93c4089980 con 0x7f93b4004850 2023-11-24T06:08:51.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.890+0000 7f93cd181700 1 --2- 172.21.15.91:0/1588530076 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93ac06aed0 0x7f93ac06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:51.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.890+0000 7f93cd181700 1 --2- 172.21.15.91:0/1588530076 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93ac06aed0 0x7f93ac06d390 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f93b8000c00 tx=0x7f93b8009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:51.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:51.894+0000 7f93be7fc700 1 -- 172.21.15.91:0/1588530076 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93c4054d80 con 0x7f93b4004850 2023-11-24T06:08:52.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.162+0000 7f93cfbe6700 1 -- 172.21.15.91:0/1588530076 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f93c80d2270 con 0x7f93b4004850 2023-11-24T06:08:52.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.163+0000 7f93be7fc700 1 -- 172.21.15.91:0/1588530076 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f93c40583a0 con 0x7f93b4004850 2023-11-24T06:08:52.167 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:52.167 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2023-11-24T06:08:52.167 INFO:teuthology.orchestra.run.smithi091.stdout: "auth_dump": [ 2023-11-24T06:08:52.167 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.168 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "mds.a", 2023-11-24T06:08:52.168 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCHPWBl/VoLMRAANt18r5ISLrCCFV/DZRefeg==", 2023-11-24T06:08:52.168 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.168 INFO:teuthology.orchestra.run.smithi091.stdout: "mds": "allow", 2023-11-24T06:08:52.168 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow *", 2023-11-24T06:08:52.168 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow *", 2023-11-24T06:08:52.168 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow *" 2023-11-24T06:08:52.169 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.169 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.169 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.169 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "mds.b", 2023-11-24T06:08:52.169 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCIPWBl4eC/ARAAEgX+/wVFtx5DZbnOtXk35A==", 2023-11-24T06:08:52.169 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.169 INFO:teuthology.orchestra.run.smithi091.stdout: "mds": "allow", 2023-11-24T06:08:52.170 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow *", 2023-11-24T06:08:52.170 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow *", 2023-11-24T06:08:52.170 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow *" 2023-11-24T06:08:52.170 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.170 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.170 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.171 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "mds.c", 2023-11-24T06:08:52.171 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCHPWBlOJQ7OBAAAb+YjyyqyYSNZUAWn9d2/g==", 2023-11-24T06:08:52.171 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.171 INFO:teuthology.orchestra.run.smithi091.stdout: "mds": "allow", 2023-11-24T06:08:52.171 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow *", 2023-11-24T06:08:52.171 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow *", 2023-11-24T06:08:52.171 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow *" 2023-11-24T06:08:52.171 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.172 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.172 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.172 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "mds.d", 2023-11-24T06:08:52.172 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCIPWBlbm0XCRAAcu/fX5wokIxgHqq4vyVblA==", 2023-11-24T06:08:52.172 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.172 INFO:teuthology.orchestra.run.smithi091.stdout: "mds": "allow", 2023-11-24T06:08:52.173 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow *", 2023-11-24T06:08:52.173 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow *", 2023-11-24T06:08:52.173 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow *" 2023-11-24T06:08:52.173 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.173 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.173 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.173 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "osd.0", 2023-11-24T06:08:52.174 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCLPWBltHiIJhAA0EAQ1DYnQEPevN1UoqneHw==", 2023-11-24T06:08:52.174 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.174 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow profile osd", 2023-11-24T06:08:52.174 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow profile osd", 2023-11-24T06:08:52.174 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow *" 2023-11-24T06:08:52.174 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.174 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.175 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.175 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "osd.1", 2023-11-24T06:08:52.175 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCOPWBlOQ+kJRAA2F7plkyuC8qRw6qMwrl2uw==", 2023-11-24T06:08:52.175 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.175 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow profile osd", 2023-11-24T06:08:52.175 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow profile osd", 2023-11-24T06:08:52.175 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow *" 2023-11-24T06:08:52.176 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.176 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.176 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.176 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "osd.2", 2023-11-24T06:08:52.176 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCRPWBlaUL7JBAAiutjTMqJ4POTynJjCvIvDg==", 2023-11-24T06:08:52.176 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.177 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow profile osd", 2023-11-24T06:08:52.177 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow profile osd", 2023-11-24T06:08:52.177 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow *" 2023-11-24T06:08:52.177 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.177 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.177 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.177 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "osd.3", 2023-11-24T06:08:52.178 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCUPWBlRsdIJBAAMgD3lTDFH3Wc7X+2ye37AQ==", 2023-11-24T06:08:52.178 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.178 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow profile osd", 2023-11-24T06:08:52.178 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow profile osd", 2023-11-24T06:08:52.178 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow *" 2023-11-24T06:08:52.178 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.178 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.179 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.179 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "osd.4", 2023-11-24T06:08:52.179 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCYPWBlE7tgMxAA13pOQ96qq/9WR6pJtmzKKw==", 2023-11-24T06:08:52.179 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.179 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow profile osd", 2023-11-24T06:08:52.179 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow profile osd", 2023-11-24T06:08:52.179 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow *" 2023-11-24T06:08:52.180 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.180 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.180 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.180 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "osd.5", 2023-11-24T06:08:52.180 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCbPWBl6o7oMhAAe++S2VE9es9jNzRRg/s0zg==", 2023-11-24T06:08:52.180 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.180 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow profile osd", 2023-11-24T06:08:52.181 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow profile osd", 2023-11-24T06:08:52.181 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow *" 2023-11-24T06:08:52.181 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.181 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.181 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.182 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "osd.6", 2023-11-24T06:08:52.182 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCePWBl4HZeMhAAefE88W6kjeS1ZMh2YzS/uw==", 2023-11-24T06:08:52.182 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.182 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow profile osd", 2023-11-24T06:08:52.182 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow profile osd", 2023-11-24T06:08:52.182 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow *" 2023-11-24T06:08:52.182 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.183 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.183 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.183 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "osd.7", 2023-11-24T06:08:52.183 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQChPWBlTGD9MRAA0lf1fKc0GVxhCvp+cckQfg==", 2023-11-24T06:08:52.183 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.183 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow profile osd", 2023-11-24T06:08:52.183 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow profile osd", 2023-11-24T06:08:52.184 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow *" 2023-11-24T06:08:52.184 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.184 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.184 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.184 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "client.0", 2023-11-24T06:08:52.184 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCIPWBlZntSDRAAdyaD3NYZ7aqLp5+7oaQH+Q==", 2023-11-24T06:08:52.184 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.185 INFO:teuthology.orchestra.run.smithi091.stdout: "mds": "allow", 2023-11-24T06:08:52.185 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow r", 2023-11-24T06:08:52.185 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow rw", 2023-11-24T06:08:52.185 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow rwx" 2023-11-24T06:08:52.185 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.185 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.185 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.186 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "client.1", 2023-11-24T06:08:52.186 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCIPWBltflUFhAAQm7p6/tZmL9YhVNjtakX6g==", 2023-11-24T06:08:52.186 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.186 INFO:teuthology.orchestra.run.smithi091.stdout: "mds": "allow", 2023-11-24T06:08:52.186 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow r", 2023-11-24T06:08:52.186 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow rw", 2023-11-24T06:08:52.187 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow rwx" 2023-11-24T06:08:52.187 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.187 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.187 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.190 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "client.2", 2023-11-24T06:08:52.190 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCIPWBlGCICExAA3BvhkJYyZ0SbJLg/PZAjpA==", 2023-11-24T06:08:52.190 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.191 INFO:teuthology.orchestra.run.smithi091.stdout: "mds": "allow", 2023-11-24T06:08:52.191 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow r", 2023-11-24T06:08:52.191 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow rw", 2023-11-24T06:08:52.191 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow rwx" 2023-11-24T06:08:52.191 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.191 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.192 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.192 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "client.3", 2023-11-24T06:08:52.192 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCIPWBlSIEHHBAAYaIDPZgQUJqKQqGcApp93Q==", 2023-11-24T06:08:52.192 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.192 INFO:teuthology.orchestra.run.smithi091.stdout: "mds": "allow", 2023-11-24T06:08:52.192 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow r", 2023-11-24T06:08:52.192 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow rw", 2023-11-24T06:08:52.193 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow rwx" 2023-11-24T06:08:52.193 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.193 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.193 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.193 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "client.admin", 2023-11-24T06:08:52.193 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCHPWBl3TT0ExAAXkxpBo9VXuRo1PDQkcAByw==", 2023-11-24T06:08:52.193 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.193 INFO:teuthology.orchestra.run.smithi091.stdout: "mds": "allow *", 2023-11-24T06:08:52.194 INFO:teuthology.orchestra.run.smithi091.stdout: "mgr": "allow *", 2023-11-24T06:08:52.194 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow *", 2023-11-24T06:08:52.194 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow *" 2023-11-24T06:08:52.194 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.194 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.194 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.195 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "client.bootstrap-mds", 2023-11-24T06:08:52.195 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCnPWBlQ6TJLxAAVCSNFH7TBD62Y1TBH2doMQ==", 2023-11-24T06:08:52.195 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.195 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow profile bootstrap-mds" 2023-11-24T06:08:52.195 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.195 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.195 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.195 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "client.bootstrap-mgr", 2023-11-24T06:08:52.196 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCnPWBl063JLxAALl99iuXaEa7yTFq8KuOroQ==", 2023-11-24T06:08:52.196 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.196 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow profile bootstrap-mgr" 2023-11-24T06:08:52.196 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.196 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.196 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.196 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "client.bootstrap-osd", 2023-11-24T06:08:52.197 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCnPWBlTLfJLxAAuItKO4dxHGyA2UllLV6hrw==", 2023-11-24T06:08:52.197 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.197 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow profile bootstrap-osd" 2023-11-24T06:08:52.197 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.197 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.197 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.197 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "client.bootstrap-rbd", 2023-11-24T06:08:52.198 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCnPWBls77JLxAARbZw1b/0tEpO6YiY4Jon/g==", 2023-11-24T06:08:52.198 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.198 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow profile bootstrap-rbd" 2023-11-24T06:08:52.198 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.198 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.198 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.198 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-11-24T06:08:52.199 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCnPWBlI8bJLxAAZ8yUyPMqrqeLg/Ui627Ylg==", 2023-11-24T06:08:52.199 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.199 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-11-24T06:08:52.199 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.199 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.199 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.199 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "client.bootstrap-rgw", 2023-11-24T06:08:52.200 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCnPWBluc7JLxAAZW+y57/zrpMZAY/Oz5P2Jw==", 2023-11-24T06:08:52.200 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.200 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow profile bootstrap-rgw" 2023-11-24T06:08:52.200 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.200 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.200 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.200 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "mgr.x", 2023-11-24T06:08:52.201 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCHPWBlg46tLRAALjHRKiAhP1of8tSavqqvxg==", 2023-11-24T06:08:52.201 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.201 INFO:teuthology.orchestra.run.smithi091.stdout: "mds": "allow *", 2023-11-24T06:08:52.201 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow profile mgr", 2023-11-24T06:08:52.201 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow *" 2023-11-24T06:08:52.201 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.201 INFO:teuthology.orchestra.run.smithi091.stdout: }, 2023-11-24T06:08:52.202 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:52.202 INFO:teuthology.orchestra.run.smithi091.stdout: "entity": "mgr.y", 2023-11-24T06:08:52.202 INFO:teuthology.orchestra.run.smithi091.stdout: "key": "AQCHPWBlDJgdKRAApNIX943pvbdHdb7Wd1qBQA==", 2023-11-24T06:08:52.202 INFO:teuthology.orchestra.run.smithi091.stdout: "caps": { 2023-11-24T06:08:52.202 INFO:teuthology.orchestra.run.smithi091.stdout: "mds": "allow *", 2023-11-24T06:08:52.202 INFO:teuthology.orchestra.run.smithi091.stdout: "mon": "allow profile mgr", 2023-11-24T06:08:52.203 INFO:teuthology.orchestra.run.smithi091.stdout: "osd": "allow *" 2023-11-24T06:08:52.203 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.203 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:52.203 INFO:teuthology.orchestra.run.smithi091.stdout: ] 2023-11-24T06:08:52.203 INFO:teuthology.orchestra.run.smithi091.stdout:} 2023-11-24T06:08:52.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.165+0000 7f93cfbe6700 1 -- 172.21.15.91:0/1588530076 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93ac06aed0 msgr2=0x7f93ac06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:52.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.165+0000 7f93cfbe6700 1 --2- 172.21.15.91:0/1588530076 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93ac06aed0 0x7f93ac06d390 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f93b8000c00 tx=0x7f93b8009030 comp rx=0 tx=0).stop 2023-11-24T06:08:52.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.165+0000 7f93cfbe6700 1 -- 172.21.15.91:0/1588530076 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93b4004850 msgr2=0x7f93c81525e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:52.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.165+0000 7f93cfbe6700 1 --2- 172.21.15.91:0/1588530076 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93b4004850 0x7f93c81525e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f93c40119a0 tx=0x7f93c4011d60 comp rx=0 tx=0).stop 2023-11-24T06:08:52.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.165+0000 7f93cfbe6700 1 -- 172.21.15.91:0/1588530076 shutdown_connections 2023-11-24T06:08:52.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.165+0000 7f93cfbe6700 1 --2- 172.21.15.91:0/1588530076 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f93ac06aed0 0x7f93ac06d390 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.165+0000 7f93cfbe6700 1 --2- 172.21.15.91:0/1588530076 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f93c80cb790 0x7f93c8160600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.165+0000 7f93cfbe6700 1 --2- 172.21.15.91:0/1588530076 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f93b4004850 0x7f93c81525e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.165+0000 7f93cfbe6700 1 --2- 172.21.15.91:0/1588530076 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f93c80cb310 0x7f93c81600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.165+0000 7f93cfbe6700 1 -- 172.21.15.91:0/1588530076 >> 172.21.15.91:0/1588530076 conn(0x7f93c80d3eb0 msgr2=0x7f93c806cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:52.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.165+0000 7f93cfbe6700 1 -- 172.21.15.91:0/1588530076 shutdown_connections 2023-11-24T06:08:52.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.165+0000 7f93cfbe6700 1 -- 172.21.15.91:0/1588530076 wait complete. 2023-11-24T06:08:52.207 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-11-24T06:08:52.328 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.324+0000 7efdd406d700 1 Processor -- start 2023-11-24T06:08:52.328 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.324+0000 7efdd406d700 1 -- start start 2023-11-24T06:08:52.330 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.325+0000 7efdd406d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7efdcc0722c0 0x7efdcc0b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:52.331 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.325+0000 7efdd406d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efdcc1440f0 0x7efdcc06e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:52.331 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.325+0000 7efdd406d700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdcc143ec0 con 0x7efdcc143ae0 2023-11-24T06:08:52.331 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.325+0000 7efdd406d700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdcc0b40a0 con 0x7efdcc0722c0 2023-11-24T06:08:52.331 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.325+0000 7efdd406d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc0b3ae0 con 0x7efdcc1440f0 2023-11-24T06:08:52.331 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.325+0000 7efdd1608700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7efdcc0722c0 0x7efdcc0b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:52.331 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.325+0000 7efdd1e09700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7efdcc143ae0 0x7efdcc071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54364/0 (socket says 172.21.15.91:54364) 2023-11-24T06:08:52.331 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.325+0000 7efdd1e09700 1 -- 172.21.15.91:0/396860367 learned_addr learned my addr 172.21.15.91:0/396860367 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:52.331 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.326+0000 7efdd0e07700 1 -- 172.21.15.91:0/396860367 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286363091 0 0) 0x7efdcc143ec0 con 0x7efdcc143ae0 2023-11-24T06:08:52.331 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.326+0000 7efdd260a700 1 --2- 172.21.15.91:0/396860367 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efdcc1440f0 0x7efdcc06e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:52.331 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.326+0000 7efdd0e07700 1 -- 172.21.15.91:0/396860367 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdb80035b0 con 0x7efdcc143ae0 2023-11-24T06:08:52.331 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.326+0000 7efdd1608700 1 -- 172.21.15.91:0/396860367 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efdcc1440f0 msgr2=0x7efdcc06e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:52.331 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.326+0000 7efdd1608700 1 --2- 172.21.15.91:0/396860367 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efdcc1440f0 0x7efdcc06e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.332 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.326+0000 7efdd1608700 1 -- 172.21.15.91:0/396860367 >> v1:172.21.15.91:6789/0 conn(0x7efdcc143ae0 legacy=0x7efdcc071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:52.332 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.326+0000 7efdd1608700 1 -- 172.21.15.91:0/396860367 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc0b3dc0 con 0x7efdcc0722c0 2023-11-24T06:08:52.332 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.326+0000 7efdd1608700 1 --2- 172.21.15.91:0/396860367 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7efdcc0722c0 0x7efdcc0b51b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009840 tx=0x7efdc8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=210d4dfd5cc1a755 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:52.332 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.326+0000 7efdd0e07700 1 -- 172.21.15.91:0/396860367 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc8011830 con 0x7efdcc0722c0 2023-11-24T06:08:52.332 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.326+0000 7efdd0e07700 1 -- 172.21.15.91:0/396860367 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc8011990 con 0x7efdcc0722c0 2023-11-24T06:08:52.332 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.326+0000 7efdd0e07700 1 -- 172.21.15.91:0/396860367 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc8011d20 con 0x7efdcc0722c0 2023-11-24T06:08:52.332 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.326+0000 7efdd406d700 1 -- 172.21.15.91:0/396860367 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7efdcc0722c0 msgr2=0x7efdcc0b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:52.332 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.326+0000 7efdd406d700 1 --2- 172.21.15.91:0/396860367 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7efdcc0722c0 0x7efdcc0b51b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009840 tx=0x7efdc8009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:52.332 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.327+0000 7efdd406d700 1 -- 172.21.15.91:0/396860367 shutdown_connections 2023-11-24T06:08:52.332 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.327+0000 7efdd406d700 1 --2- 172.21.15.91:0/396860367 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efdcc1440f0 0x7efdcc06e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.332 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.327+0000 7efdd406d700 1 --2- 172.21.15.91:0/396860367 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7efdcc0722c0 0x7efdcc0b51b0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.332 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.327+0000 7efdd406d700 1 -- 172.21.15.91:0/396860367 >> 172.21.15.91:0/396860367 conn(0x7efdcc0b2740 msgr2=0x7efdcc0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:52.333 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.327+0000 7efdd406d700 1 -- 172.21.15.91:0/396860367 shutdown_connections 2023-11-24T06:08:52.333 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.327+0000 7efdd406d700 1 -- 172.21.15.91:0/396860367 wait complete. 2023-11-24T06:08:52.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.328+0000 7efdd406d700 1 Processor -- start 2023-11-24T06:08:52.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.328+0000 7efdd406d700 1 -- start start 2023-11-24T06:08:52.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.329+0000 7efdd406d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdcc0722c0 0x7efdcc0cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:52.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.329+0000 7efdd406d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7efdcc143ae0 0x7efdcc1418d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:52.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.329+0000 7efdd406d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efdcc1440f0 0x7efdcc141e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:52.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.329+0000 7efdd406d700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc0cc6e0 con 0x7efdcc143ae0 2023-11-24T06:08:52.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.329+0000 7efdd406d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc1f7410 con 0x7efdcc0722c0 2023-11-24T06:08:52.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.329+0000 7efdd406d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdcc1f76f0 con 0x7efdcc1440f0 2023-11-24T06:08:52.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.329+0000 7efdd1608700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7efdcc143ae0 0x7efdcc1418d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:52.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.329+0000 7efdd1608700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7efdcc143ae0 0x7efdcc1418d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59438/0 (socket says 172.21.15.91:59438) 2023-11-24T06:08:52.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.329+0000 7efdd1608700 1 -- 172.21.15.91:0/942735715 learned_addr learned my addr 172.21.15.91:0/942735715 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:52.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.329+0000 7efdd260a700 1 --2- 172.21.15.91:0/942735715 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efdcc1440f0 0x7efdcc141e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:52.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.329+0000 7efdd1e09700 1 --2- 172.21.15.91:0/942735715 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdcc0722c0 0x7efdcc0cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:52.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.330+0000 7efdd1608700 1 -- 172.21.15.91:0/942735715 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efdcc1440f0 msgr2=0x7efdcc141e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:52.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.330+0000 7efdd1608700 1 --2- 172.21.15.91:0/942735715 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efdcc1440f0 0x7efdcc141e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.330+0000 7efdd1608700 1 -- 172.21.15.91:0/942735715 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdcc0722c0 msgr2=0x7efdcc0cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:52.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.330+0000 7efdd1608700 1 --2- 172.21.15.91:0/942735715 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdcc0722c0 0x7efdcc0cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.330+0000 7efdd1608700 1 -- 172.21.15.91:0/942735715 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc80094a0 con 0x7efdcc143ae0 2023-11-24T06:08:52.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.330+0000 7efdd1608700 1 --2- 172.21.15.91:0/942735715 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7efdcc143ae0 0x7efdcc1418d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009fd0 tx=0x7efdc8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:52.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.330+0000 7efdc27fc700 1 -- 172.21.15.91:0/942735715 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc80036a0 con 0x7efdcc143ae0 2023-11-24T06:08:52.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.330+0000 7efdc27fc700 1 -- 172.21.15.91:0/942735715 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc801f070 con 0x7efdcc143ae0 2023-11-24T06:08:52.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.330+0000 7efdd406d700 1 -- 172.21.15.91:0/942735715 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdcc1f79d0 con 0x7efdcc143ae0 2023-11-24T06:08:52.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.330+0000 7efdc27fc700 1 -- 172.21.15.91:0/942735715 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc8003a60 con 0x7efdcc143ae0 2023-11-24T06:08:52.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.330+0000 7efdd406d700 1 -- 172.21.15.91:0/942735715 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdcc1353d0 con 0x7efdcc143ae0 2023-11-24T06:08:52.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.331+0000 7efdd406d700 1 -- 172.21.15.91:0/942735715 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdcc1f8020 con 0x7efdcc143ae0 2023-11-24T06:08:52.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.331+0000 7efdc27fc700 1 -- 172.21.15.91:0/942735715 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7efdc8003bc0 con 0x7efdcc143ae0 2023-11-24T06:08:52.338 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.332+0000 7efdc27fc700 1 --2- 172.21.15.91:0/942735715 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7efdb006aed0 0x7efdb006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:52.338 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.332+0000 7efdc27fc700 1 -- 172.21.15.91:0/942735715 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efdc8021070 con 0x7efdcc143ae0 2023-11-24T06:08:52.338 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.332+0000 7efdd1e09700 1 --2- 172.21.15.91:0/942735715 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7efdb006aed0 0x7efdb006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:52.338 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.332+0000 7efdd1e09700 1 --2- 172.21.15.91:0/942735715 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7efdb006aed0 0x7efdb006d390 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efdbc007fd0 tx=0x7efdbc007dc0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:52.340 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.336+0000 7efdc27fc700 1 -- 172.21.15.91:0/942735715 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efdc8055e50 con 0x7efdcc143ae0 2023-11-24T06:08:52.605 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.601+0000 7efdd406d700 1 -- 172.21.15.91:0/942735715 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7efdcc15e200 con 0x7efdcc143ae0 2023-11-24T06:08:52.607 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.603+0000 7efdc27fc700 1 -- 172.21.15.91:0/942735715 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7efdc8015020 con 0x7efdcc143ae0 2023-11-24T06:08:52.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.605+0000 7efdd406d700 1 -- 172.21.15.91:0/942735715 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7efdb006aed0 msgr2=0x7efdb006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:52.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.605+0000 7efdd406d700 1 --2- 172.21.15.91:0/942735715 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7efdb006aed0 0x7efdb006d390 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efdbc007fd0 tx=0x7efdbc007dc0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.605+0000 7efdd406d700 1 -- 172.21.15.91:0/942735715 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7efdcc143ae0 msgr2=0x7efdcc1418d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:52.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.605+0000 7efdd406d700 1 --2- 172.21.15.91:0/942735715 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7efdcc143ae0 0x7efdcc1418d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009fd0 tx=0x7efdc8004750 comp rx=0 tx=0).stop 2023-11-24T06:08:52.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.606+0000 7efdd406d700 1 -- 172.21.15.91:0/942735715 shutdown_connections 2023-11-24T06:08:52.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.606+0000 7efdd406d700 1 --2- 172.21.15.91:0/942735715 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7efdb006aed0 0x7efdb006d390 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.610 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.606+0000 7efdd406d700 1 --2- 172.21.15.91:0/942735715 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7efdcc0722c0 0x7efdcc0cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.606+0000 7efdd406d700 1 --2- 172.21.15.91:0/942735715 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7efdcc1440f0 0x7efdcc141e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.606+0000 7efdd406d700 1 --2- 172.21.15.91:0/942735715 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7efdcc143ae0 0x7efdcc1418d0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.606+0000 7efdd406d700 1 -- 172.21.15.91:0/942735715 >> 172.21.15.91:0/942735715 conn(0x7efdcc0b2740 msgr2=0x7efdcc071bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:52.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.606+0000 7efdd406d700 1 -- 172.21.15.91:0/942735715 shutdown_connections 2023-11-24T06:08:52.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.606+0000 7efdd406d700 1 -- 172.21.15.91:0/942735715 wait complete. 2023-11-24T06:08:52.621 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-11-24T06:08:52.742 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.738+0000 7f6824632700 1 Processor -- start 2023-11-24T06:08:52.742 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.738+0000 7f6824632700 1 -- start start 2023-11-24T06:08:52.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.739+0000 7f6824632700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f681c073280 0x7f681c0b01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:52.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.739+0000 7f6824632700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f681c161a00 0x7f681c0726a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:52.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.739+0000 7f6824632700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681c0ccd20 con 0x7f681c161620 2023-11-24T06:08:52.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.739+0000 7f6824632700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c06b100 con 0x7f681c073280 2023-11-24T06:08:52.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.739+0000 7f6824632700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c06b3e0 con 0x7f681c161a00 2023-11-24T06:08:52.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.739+0000 7f6821bcd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f681c073280 0x7f681c0b01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:52.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.739+0000 7f6821bcd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f681c073280 0x7f681c0b01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59446/0 (socket says 172.21.15.91:59446) 2023-11-24T06:08:52.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.740+0000 7f6821bcd700 1 -- 172.21.15.91:0/2157371850 learned_addr learned my addr 172.21.15.91:0/2157371850 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:52.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.740+0000 7f6822bcf700 1 --2- 172.21.15.91:0/2157371850 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f681c161a00 0x7f681c0726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:52.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.740+0000 7f6821bcd700 1 -- 172.21.15.91:0/2157371850 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f681c161a00 msgr2=0x7f681c0726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:52.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.740+0000 7f6821bcd700 1 --2- 172.21.15.91:0/2157371850 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f681c161a00 0x7f681c0726a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.740+0000 7f6821bcd700 1 -- 172.21.15.91:0/2157371850 >> v1:172.21.15.94:6789/0 conn(0x7f681c161620 legacy=0x7f681c066ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:52.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.740+0000 7f6821bcd700 1 -- 172.21.15.91:0/2157371850 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c06b6c0 con 0x7f681c073280 2023-11-24T06:08:52.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.740+0000 7f6821bcd700 1 --2- 172.21.15.91:0/2157371850 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f681c073280 0x7f681c0b01b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f68180098f0 tx=0x7f6818009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da1279f67dc623d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:52.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.740+0000 7f68213cc700 1 -- 172.21.15.91:0/2157371850 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681800b8a0 con 0x7f681c073280 2023-11-24T06:08:52.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.740+0000 7f68213cc700 1 -- 172.21.15.91:0/2157371850 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681800ba00 con 0x7f681c073280 2023-11-24T06:08:52.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.740+0000 7f68213cc700 1 -- 172.21.15.91:0/2157371850 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681800bd90 con 0x7f681c073280 2023-11-24T06:08:52.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.741+0000 7f6824632700 1 -- 172.21.15.91:0/2157371850 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f681c073280 msgr2=0x7f681c0b01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:52.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.741+0000 7f6824632700 1 --2- 172.21.15.91:0/2157371850 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f681c073280 0x7f681c0b01b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f68180098f0 tx=0x7f6818009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:52.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.741+0000 7f6824632700 1 -- 172.21.15.91:0/2157371850 shutdown_connections 2023-11-24T06:08:52.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.741+0000 7f6824632700 1 --2- 172.21.15.91:0/2157371850 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f681c161a00 0x7f681c0726a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.741+0000 7f6824632700 1 --2- 172.21.15.91:0/2157371850 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f681c073280 0x7f681c0b01b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.741+0000 7f6824632700 1 -- 172.21.15.91:0/2157371850 >> 172.21.15.91:0/2157371850 conn(0x7f681c0b50e0 msgr2=0x7f681c0b57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:52.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.741+0000 7f6824632700 1 -- 172.21.15.91:0/2157371850 shutdown_connections 2023-11-24T06:08:52.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.741+0000 7f6824632700 1 -- 172.21.15.91:0/2157371850 wait complete. 2023-11-24T06:08:52.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.742+0000 7f6824632700 1 Processor -- start 2023-11-24T06:08:52.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.742+0000 7f6824632700 1 -- start start 2023-11-24T06:08:52.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6824632700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f681c073280 0x7f681c1567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:52.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6824632700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f681c161620 0x7f681c0ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:52.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6824632700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f681c161a00 0x7f681c0cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:52.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6824632700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c15bb90 con 0x7f681c161a00 2023-11-24T06:08:52.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6824632700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c15af80 con 0x7f681c073280 2023-11-24T06:08:52.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6824632700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c15b260 con 0x7f681c161620 2023-11-24T06:08:52.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6822bcf700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f681c161a00 0x7f681c0cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:52.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6822bcf700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f681c161a00 0x7f681c0cd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59454/0 (socket says 172.21.15.91:59454) 2023-11-24T06:08:52.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6822bcf700 1 -- 172.21.15.91:0/3225705880 learned_addr learned my addr 172.21.15.91:0/3225705880 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:52.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6821bcd700 1 --2- 172.21.15.91:0/3225705880 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f681c161620 0x7f681c0ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:52.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f68223ce700 1 --2- 172.21.15.91:0/3225705880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f681c073280 0x7f681c1567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:52.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6822bcf700 1 -- 172.21.15.91:0/3225705880 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f681c161620 msgr2=0x7f681c0ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:52.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6822bcf700 1 --2- 172.21.15.91:0/3225705880 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f681c161620 0x7f681c0ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6822bcf700 1 -- 172.21.15.91:0/3225705880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f681c073280 msgr2=0x7f681c1567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:52.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6822bcf700 1 --2- 172.21.15.91:0/3225705880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f681c073280 0x7f681c1567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:52.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6822bcf700 1 -- 172.21.15.91:0/3225705880 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6818009550 con 0x7f681c161a00 2023-11-24T06:08:52.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f68223ce700 1 --2- 172.21.15.91:0/3225705880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f681c073280 0x7f681c1567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:52.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6821bcd700 1 --2- 172.21.15.91:0/3225705880 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f681c161620 0x7f681c0ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:52.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6822bcf700 1 --2- 172.21.15.91:0/3225705880 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f681c161a00 0x7f681c0cd390 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6814007e60 tx=0x7f681400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:52.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6812ffd700 1 -- 172.21.15.91:0/3225705880 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681400ede0 con 0x7f681c161a00 2023-11-24T06:08:52.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6812ffd700 1 -- 172.21.15.91:0/3225705880 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6814014420 con 0x7f681c161a00 2023-11-24T06:08:52.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6824632700 1 -- 172.21.15.91:0/3225705880 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c15b540 con 0x7f681c161a00 2023-11-24T06:08:52.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6812ffd700 1 -- 172.21.15.91:0/3225705880 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68140147d0 con 0x7f681c161a00 2023-11-24T06:08:52.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.743+0000 7f6824632700 1 -- 172.21.15.91:0/3225705880 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c156d30 con 0x7f681c161a00 2023-11-24T06:08:52.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.744+0000 7f6824632700 1 -- 172.21.15.91:0/3225705880 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c15bd00 con 0x7f681c161a00 2023-11-24T06:08:52.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.745+0000 7f6812ffd700 1 -- 172.21.15.91:0/3225705880 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f6814014930 con 0x7f681c161a00 2023-11-24T06:08:52.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.745+0000 7f6812ffd700 1 --2- 172.21.15.91:0/3225705880 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f680006aed0 0x7f680006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:52.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.745+0000 7f6812ffd700 1 -- 172.21.15.91:0/3225705880 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6814089950 con 0x7f681c161a00 2023-11-24T06:08:52.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.745+0000 7f68223ce700 1 --2- 172.21.15.91:0/3225705880 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f680006aed0 0x7f680006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:52.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.746+0000 7f68223ce700 1 --2- 172.21.15.91:0/3225705880 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f680006aed0 0x7f680006d390 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f680c000f60 tx=0x7f680c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:52.753 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:52.750+0000 7f6812ffd700 1 -- 172.21.15.91:0/3225705880 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6814054de0 con 0x7f681c161a00 2023-11-24T06:08:53.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.018+0000 7f6824632700 1 -- 172.21.15.91:0/3225705880 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f681c068670 con 0x7f681c161a00 2023-11-24T06:08:53.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.020+0000 7f6812ffd700 1 -- 172.21.15.91:0/3225705880 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f6814058400 con 0x7f681c161a00 2023-11-24T06:08:53.027 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.023+0000 7f6824632700 1 -- 172.21.15.91:0/3225705880 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f680006aed0 msgr2=0x7f680006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:53.027 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.023+0000 7f6824632700 1 --2- 172.21.15.91:0/3225705880 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f680006aed0 0x7f680006d390 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f680c000f60 tx=0x7f680c009040 comp rx=0 tx=0).stop 2023-11-24T06:08:53.027 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.023+0000 7f6824632700 1 -- 172.21.15.91:0/3225705880 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f681c161a00 msgr2=0x7f681c0cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:53.027 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.023+0000 7f6824632700 1 --2- 172.21.15.91:0/3225705880 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f681c161a00 0x7f681c0cd390 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6814007e60 tx=0x7f681400e3c0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.027 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.023+0000 7f6824632700 1 -- 172.21.15.91:0/3225705880 shutdown_connections 2023-11-24T06:08:53.027 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.023+0000 7f6824632700 1 --2- 172.21.15.91:0/3225705880 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f680006aed0 0x7f680006d390 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.023+0000 7f6824632700 1 --2- 172.21.15.91:0/3225705880 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f681c073280 0x7f681c1567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.023+0000 7f6824632700 1 --2- 172.21.15.91:0/3225705880 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f681c161a00 0x7f681c0cd390 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.023+0000 7f6824632700 1 --2- 172.21.15.91:0/3225705880 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f681c161620 0x7f681c0ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.023+0000 7f6824632700 1 -- 172.21.15.91:0/3225705880 >> 172.21.15.91:0/3225705880 conn(0x7f681c0b50e0 msgr2=0x7f681c06c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:53.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.023+0000 7f6824632700 1 -- 172.21.15.91:0/3225705880 shutdown_connections 2023-11-24T06:08:53.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.023+0000 7f6824632700 1 -- 172.21.15.91:0/3225705880 wait complete. 2023-11-24T06:08:53.038 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-11-24T06:08:53.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.156+0000 7f576a9b7700 1 Processor -- start 2023-11-24T06:08:53.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.156+0000 7f576a9b7700 1 -- start start 2023-11-24T06:08:53.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.157+0000 7f576a9b7700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f576415bde0 0x7f576415c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.157+0000 7f576a9b7700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57640d0a90 con 0x7f57640ce5c0 2023-11-24T06:08:53.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.157+0000 7f576a9b7700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f576406ee10 con 0x7f57640cf2b0 2023-11-24T06:08:53.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.157+0000 7f576a9b7700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f576406f0f0 con 0x7f576415bde0 2023-11-24T06:08:53.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.157+0000 7f5768f54700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f57640ce5c0 0x7f576406fbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54394/0 (socket says 172.21.15.91:54394) 2023-11-24T06:08:53.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.157+0000 7f5768f54700 1 -- 172.21.15.91:0/1777706498 learned_addr learned my addr 172.21.15.91:0/1777706498 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:53.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.157+0000 7f5763fff700 1 --2- 172.21.15.91:0/1777706498 >> v2:172.21.15.94:3300/0 conn(0x7f576415bde0 0x7f576415c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:53.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5762ffd700 1 -- 172.21.15.91:0/1777706498 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660472678 0 0) 0x7f57640d0a90 con 0x7f57640ce5c0 2023-11-24T06:08:53.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5762ffd700 1 -- 172.21.15.91:0/1777706498 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f574c0035b0 con 0x7f57640ce5c0 2023-11-24T06:08:53.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5762ffd700 1 -- 172.21.15.91:0/1777706498 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2609336408 0 0) 0x7f574c0035b0 con 0x7f57640ce5c0 2023-11-24T06:08:53.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5762ffd700 1 -- 172.21.15.91:0/1777706498 >> v2:172.21.15.94:3300/0 conn(0x7f576415bde0 msgr2=0x7f576415c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:53.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5762ffd700 1 --2- 172.21.15.91:0/1777706498 >> v2:172.21.15.94:3300/0 conn(0x7f576415bde0 0x7f576415c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5762ffd700 1 -- 172.21.15.91:0/1777706498 >> v1:172.21.15.94:6789/0 conn(0x7f57640cf2b0 legacy=0x7f57640d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:53.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5762ffd700 1 -- 172.21.15.91:0/1777706498 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576406f3d0 con 0x7f57640ce5c0 2023-11-24T06:08:53.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5762ffd700 1 -- 172.21.15.91:0/1777706498 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f575c0019f0 con 0x7f57640ce5c0 2023-11-24T06:08:53.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5762ffd700 1 -- 172.21.15.91:0/1777706498 >> v1:172.21.15.91:6789/0 conn(0x7f57640ce5c0 legacy=0x7f576406fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:53.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5762ffd700 1 --2- 172.21.15.91:0/1777706498 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f574c003a00 0x7f574c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5762ffd700 1 --2- 172.21.15.91:0/1777706498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f574c004850 0x7f574c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5762ffd700 1 --2- 172.21.15.91:0/1777706498 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f574c005210 0x7f574c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5762ffd700 1 -- 172.21.15.91:0/1777706498 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576406f0f0 con 0x7f574c005210 2023-11-24T06:08:53.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5762ffd700 1 -- 172.21.15.91:0/1777706498 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c00c1b0 con 0x7f574c004850 2023-11-24T06:08:53.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5762ffd700 1 -- 172.21.15.91:0/1777706498 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574c00c370 con 0x7f574c003a00 2023-11-24T06:08:53.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5768f54700 1 --2- 172.21.15.91:0/1777706498 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f574c005210 0x7f574c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:53.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5763fff700 1 --2- 172.21.15.91:0/1777706498 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f574c003a00 0x7f574c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:53.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f57637fe700 1 --2- 172.21.15.91:0/1777706498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f574c004850 0x7f574c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:53.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5768f54700 1 -- 172.21.15.91:0/1777706498 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f574c003a00 msgr2=0x7f574c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:53.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5768f54700 1 --2- 172.21.15.91:0/1777706498 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f574c003a00 0x7f574c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5768f54700 1 -- 172.21.15.91:0/1777706498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f574c004850 msgr2=0x7f574c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:53.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5768f54700 1 --2- 172.21.15.91:0/1777706498 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f574c004850 0x7f574c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5768f54700 1 -- 172.21.15.91:0/1777706498 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c0035b0 con 0x7f574c005210 2023-11-24T06:08:53.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.158+0000 7f5768f54700 1 -- 172.21.15.91:0/1777706498 reap_dead start 2023-11-24T06:08:53.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.159+0000 7f5768f54700 1 --2- 172.21.15.91:0/1777706498 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f574c005210 0x7f574c00ba10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f575c00a830 tx=0x7f574c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f843d5f97a1e576a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:53.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.159+0000 7f5762ffd700 1 -- 172.21.15.91:0/1777706498 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c00a2f0 con 0x7f574c005210 2023-11-24T06:08:53.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.159+0000 7f5762ffd700 1 -- 172.21.15.91:0/1777706498 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575c012620 con 0x7f574c005210 2023-11-24T06:08:53.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.159+0000 7f5762ffd700 1 -- 172.21.15.91:0/1777706498 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c0129d0 con 0x7f574c005210 2023-11-24T06:08:53.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.159+0000 7f576a9b7700 1 -- 172.21.15.91:0/1777706498 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f574c005210 msgr2=0x7f574c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:53.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.159+0000 7f576a9b7700 1 --2- 172.21.15.91:0/1777706498 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f574c005210 0x7f574c00ba10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f575c00a830 tx=0x7f574c005880 comp rx=0 tx=0).stop 2023-11-24T06:08:53.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.159+0000 7f576a9b7700 1 -- 172.21.15.91:0/1777706498 shutdown_connections 2023-11-24T06:08:53.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.159+0000 7f576a9b7700 1 --2- 172.21.15.91:0/1777706498 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f574c005210 0x7f574c00ba10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.159+0000 7f576a9b7700 1 -- 172.21.15.91:0/1777706498 >> 172.21.15.91:0/1777706498 conn(0x7f57640b3720 msgr2=0x7f57640b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:53.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.159+0000 7f576a9b7700 1 -- 172.21.15.91:0/1777706498 shutdown_connections 2023-11-24T06:08:53.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.159+0000 7f576a9b7700 1 -- 172.21.15.91:0/1777706498 wait complete. 2023-11-24T06:08:53.170 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.161+0000 7f576a9b7700 1 Processor -- start 2023-11-24T06:08:53.170 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.161+0000 7f576a9b7700 1 -- start start 2023-11-24T06:08:53.170 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f576a9b7700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f574c005210 0x7f57641e58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.170 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f576a9b7700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f57640c7f30 0x7f57641e5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f576a9b7700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57640b6b50 0x7f57640b6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f576a9b7700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57640ca420 con 0x7f574c005210 2023-11-24T06:08:53.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f576a9b7700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57640c9810 con 0x7f57640b6b50 2023-11-24T06:08:53.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f576a9b7700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57640c9af0 con 0x7f57640c7f30 2023-11-24T06:08:53.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f5763fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f574c005210 0x7f57641e58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:53.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f5763fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f574c005210 0x7f57641e58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59468/0 (socket says 172.21.15.91:59468) 2023-11-24T06:08:53.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f5763fff700 1 -- 172.21.15.91:0/3555937133 learned_addr learned my addr 172.21.15.91:0/3555937133 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:53.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f57637fe700 1 --2- 172.21.15.91:0/3555937133 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f57640c7f30 0x7f57641e5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:53.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f5768f54700 1 --2- 172.21.15.91:0/3555937133 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57640b6b50 0x7f57640b6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:53.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f5763fff700 1 -- 172.21.15.91:0/3555937133 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f57640c7f30 msgr2=0x7f57641e5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:53.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f5763fff700 1 --2- 172.21.15.91:0/3555937133 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f57640c7f30 0x7f57641e5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f5763fff700 1 -- 172.21.15.91:0/3555937133 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57640b6b50 msgr2=0x7f57640b6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:53.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f5763fff700 1 --2- 172.21.15.91:0/3555937133 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57640b6b50 0x7f57640b6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f5763fff700 1 -- 172.21.15.91:0/3555937133 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57640c9dd0 con 0x7f574c005210 2023-11-24T06:08:53.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f57637fe700 1 --2- 172.21.15.91:0/3555937133 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f57640c7f30 0x7f57641e5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:53.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f5763fff700 1 --2- 172.21.15.91:0/3555937133 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f574c005210 0x7f57641e58e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f57540119c0 tx=0x7f5754011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:53.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f5760ff9700 1 -- 172.21.15.91:0/3555937133 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575400f940 con 0x7f574c005210 2023-11-24T06:08:53.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.162+0000 7f5760ff9700 1 -- 172.21.15.91:0/3555937133 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575400faa0 con 0x7f574c005210 2023-11-24T06:08:53.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.163+0000 7f576a9b7700 1 -- 172.21.15.91:0/3555937133 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57640ca0b0 con 0x7f574c005210 2023-11-24T06:08:53.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.163+0000 7f5760ff9700 1 -- 172.21.15.91:0/3555937133 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575400fe50 con 0x7f574c005210 2023-11-24T06:08:53.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.163+0000 7f576a9b7700 1 -- 172.21.15.91:0/3555937133 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57640b4810 con 0x7f574c005210 2023-11-24T06:08:53.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.163+0000 7f576a9b7700 1 -- 172.21.15.91:0/3555937133 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57640ca700 con 0x7f574c005210 2023-11-24T06:08:53.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.164+0000 7f5760ff9700 1 -- 172.21.15.91:0/3555937133 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5754014440 con 0x7f574c005210 2023-11-24T06:08:53.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.164+0000 7f5760ff9700 1 --2- 172.21.15.91:0/3555937133 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f575006aed0 0x7f575006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.164+0000 7f5760ff9700 1 -- 172.21.15.91:0/3555937133 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5754089af0 con 0x7f574c005210 2023-11-24T06:08:53.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.164+0000 7f57637fe700 1 --2- 172.21.15.91:0/3555937133 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f575006aed0 0x7f575006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:53.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.165+0000 7f57637fe700 1 --2- 172.21.15.91:0/3555937133 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f575006aed0 0x7f575006d390 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f57640c8f20 tx=0x7f5758009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:53.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.169+0000 7f5760ff9700 1 -- 172.21.15.91:0/3555937133 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5754054e70 con 0x7f574c005210 2023-11-24T06:08:53.440 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.436+0000 7f576a9b7700 1 -- 172.21.15.91:0/3555937133 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f5764068670 con 0x7f574c005210 2023-11-24T06:08:53.442 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.438+0000 7f5760ff9700 1 -- 172.21.15.91:0/3555937133 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f57540220a0 con 0x7f574c005210 2023-11-24T06:08:53.444 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.440+0000 7f576a9b7700 1 -- 172.21.15.91:0/3555937133 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f575006aed0 msgr2=0x7f575006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:53.444 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.440+0000 7f576a9b7700 1 --2- 172.21.15.91:0/3555937133 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f575006aed0 0x7f575006d390 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f57640c8f20 tx=0x7f5758009040 comp rx=0 tx=0).stop 2023-11-24T06:08:53.444 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.440+0000 7f576a9b7700 1 -- 172.21.15.91:0/3555937133 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f574c005210 msgr2=0x7f57641e58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:53.444 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.440+0000 7f576a9b7700 1 --2- 172.21.15.91:0/3555937133 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f574c005210 0x7f57641e58e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f57540119c0 tx=0x7f5754011d80 comp rx=0 tx=0).stop 2023-11-24T06:08:53.444 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.440+0000 7f576a9b7700 1 -- 172.21.15.91:0/3555937133 shutdown_connections 2023-11-24T06:08:53.444 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.440+0000 7f576a9b7700 1 --2- 172.21.15.91:0/3555937133 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f575006aed0 0x7f575006d390 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.445 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.440+0000 7f576a9b7700 1 --2- 172.21.15.91:0/3555937133 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f57640b6b50 0x7f57640b6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.445 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.440+0000 7f576a9b7700 1 --2- 172.21.15.91:0/3555937133 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f574c005210 0x7f57641e58e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.445 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.440+0000 7f576a9b7700 1 --2- 172.21.15.91:0/3555937133 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f57640c7f30 0x7f57641e5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.445 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.440+0000 7f576a9b7700 1 -- 172.21.15.91:0/3555937133 >> 172.21.15.91:0/3555937133 conn(0x7f57640b3720 msgr2=0x7f57640cc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:53.445 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.440+0000 7f576a9b7700 1 -- 172.21.15.91:0/3555937133 shutdown_connections 2023-11-24T06:08:53.445 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.440+0000 7f576a9b7700 1 -- 172.21.15.91:0/3555937133 wait complete. 2023-11-24T06:08:53.456 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-11-24T06:08:53.577 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.573+0000 7fd2aa4a1700 1 Processor -- start 2023-11-24T06:08:53.577 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.573+0000 7fd2aa4a1700 1 -- start start 2023-11-24T06:08:53.579 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.574+0000 7fd2aa4a1700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd2a40b4150 0x7fd2a406de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.579 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.574+0000 7fd2aa4a1700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd2a40b5ab0 0x7fd2a406e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.579 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.574+0000 7fd2aa4a1700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a4071f50 con 0x7fd2a406b620 2023-11-24T06:08:53.579 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.574+0000 7fd2aa4a1700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a4072230 con 0x7fd2a40b5ab0 2023-11-24T06:08:53.579 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.574+0000 7fd2aa4a1700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a4072510 con 0x7fd2a40b4150 2023-11-24T06:08:53.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.574+0000 7fd2a8a3e700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd2a40b5ab0 0x7fd2a406e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:53.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.574+0000 7fd2a8a3e700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd2a40b5ab0 0x7fd2a406e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59480/0 (socket says 172.21.15.91:59480) 2023-11-24T06:08:53.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.574+0000 7fd2a8a3e700 1 -- 172.21.15.91:0/3721840706 learned_addr learned my addr 172.21.15.91:0/3721840706 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:53.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.574+0000 7fd2a37fe700 1 --2- 172.21.15.91:0/3721840706 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd2a40b4150 0x7fd2a406de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:53.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.574+0000 7fd2a8a3e700 1 -- 172.21.15.91:0/3721840706 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd2a40b4150 msgr2=0x7fd2a406de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:53.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.574+0000 7fd2a8a3e700 1 --2- 172.21.15.91:0/3721840706 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd2a40b4150 0x7fd2a406de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.574+0000 7fd2a8a3e700 1 -- 172.21.15.91:0/3721840706 >> v1:172.21.15.94:6789/0 conn(0x7fd2a406b620 legacy=0x7fd2a40b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:53.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.574+0000 7fd2a8a3e700 1 -- 172.21.15.91:0/3721840706 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a40727f0 con 0x7fd2a40b5ab0 2023-11-24T06:08:53.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.574+0000 7fd2a37fe700 1 --2- 172.21.15.91:0/3721840706 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd2a40b4150 0x7fd2a406de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:53.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.574+0000 7fd2a8a3e700 1 --2- 172.21.15.91:0/3721840706 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2a40b5ab0 0x7fd2a406e390 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd29c009e50 tx=0x7fd29c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a057b28db9bab710 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:53.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.575+0000 7fd2a2ffd700 1 -- 172.21.15.91:0/3721840706 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c00e070 con 0x7fd2a40b5ab0 2023-11-24T06:08:53.581 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.575+0000 7fd2a2ffd700 1 -- 172.21.15.91:0/3721840706 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c003a20 con 0x7fd2a40b5ab0 2023-11-24T06:08:53.581 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.575+0000 7fd2a2ffd700 1 -- 172.21.15.91:0/3721840706 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c003dd0 con 0x7fd2a40b5ab0 2023-11-24T06:08:53.581 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.575+0000 7fd2aa4a1700 1 -- 172.21.15.91:0/3721840706 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2a40b5ab0 msgr2=0x7fd2a406e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:53.581 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.575+0000 7fd2aa4a1700 1 --2- 172.21.15.91:0/3721840706 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2a40b5ab0 0x7fd2a406e390 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd29c009e50 tx=0x7fd29c00b330 comp rx=0 tx=0).stop 2023-11-24T06:08:53.581 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.575+0000 7fd2aa4a1700 1 -- 172.21.15.91:0/3721840706 shutdown_connections 2023-11-24T06:08:53.581 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.575+0000 7fd2aa4a1700 1 --2- 172.21.15.91:0/3721840706 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2a40b5ab0 0x7fd2a406e390 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.581 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.575+0000 7fd2aa4a1700 1 --2- 172.21.15.91:0/3721840706 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd2a40b4150 0x7fd2a406de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.581 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.575+0000 7fd2aa4a1700 1 -- 172.21.15.91:0/3721840706 >> 172.21.15.91:0/3721840706 conn(0x7fd2a40b2740 msgr2=0x7fd2a40b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:53.581 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.575+0000 7fd2aa4a1700 1 -- 172.21.15.91:0/3721840706 shutdown_connections 2023-11-24T06:08:53.581 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.575+0000 7fd2aa4a1700 1 -- 172.21.15.91:0/3721840706 wait complete. 2023-11-24T06:08:53.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.577+0000 7fd2aa4a1700 1 Processor -- start 2023-11-24T06:08:53.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.577+0000 7fd2aa4a1700 1 -- start start 2023-11-24T06:08:53.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2aa4a1700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2a406b620 0x7fd2a41f4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2aa4a1700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2a40b4150 0x7fd2a40cf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2aa4a1700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd2a40b5ab0 0x7fd2a41f7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2aa4a1700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a40c9ad0 con 0x7fd2a406b620 2023-11-24T06:08:53.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2aa4a1700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a41f4c60 con 0x7fd2a40b4150 2023-11-24T06:08:53.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2aa4a1700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a41f3c60 con 0x7fd2a40b5ab0 2023-11-24T06:08:53.584 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2a3fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2a406b620 0x7fd2a41f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:53.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2a3fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2a406b620 0x7fd2a41f4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59482/0 (socket says 172.21.15.91:59482) 2023-11-24T06:08:53.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2a3fff700 1 -- 172.21.15.91:0/1130612962 learned_addr learned my addr 172.21.15.91:0/1130612962 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:53.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2a37fe700 1 --2- 172.21.15.91:0/1130612962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2a40b4150 0x7fd2a40cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:53.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2a8a3e700 1 --2- 172.21.15.91:0/1130612962 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd2a40b5ab0 0x7fd2a41f7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:53.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2a3fff700 1 -- 172.21.15.91:0/1130612962 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd2a40b5ab0 msgr2=0x7fd2a41f7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:53.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2a3fff700 1 --2- 172.21.15.91:0/1130612962 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd2a40b5ab0 0x7fd2a41f7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2a3fff700 1 -- 172.21.15.91:0/1130612962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2a40b4150 msgr2=0x7fd2a40cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:53.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2a3fff700 1 --2- 172.21.15.91:0/1130612962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2a40b4150 0x7fd2a40cf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2a3fff700 1 -- 172.21.15.91:0/1130612962 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29c009b30 con 0x7fd2a406b620 2023-11-24T06:08:53.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2a3fff700 1 --2- 172.21.15.91:0/1130612962 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2a406b620 0x7fd2a41f4720 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd29400c4d0 tx=0x7fd29400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:53.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2a0ff9700 1 -- 172.21.15.91:0/1130612962 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd294011790 con 0x7fd2a406b620 2023-11-24T06:08:53.585 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2a0ff9700 1 -- 172.21.15.91:0/1130612962 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2940118f0 con 0x7fd2a406b620 2023-11-24T06:08:53.586 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2a0ff9700 1 -- 172.21.15.91:0/1130612962 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd294011ca0 con 0x7fd2a406b620 2023-11-24T06:08:53.586 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.578+0000 7fd2aa4a1700 1 -- 172.21.15.91:0/1130612962 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a41f7cd0 con 0x7fd2a406b620 2023-11-24T06:08:53.586 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.579+0000 7fd2aa4a1700 1 -- 172.21.15.91:0/1130612962 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a41f3f20 con 0x7fd2a406b620 2023-11-24T06:08:53.586 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.579+0000 7fd2aa4a1700 1 -- 172.21.15.91:0/1130612962 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a4068670 con 0x7fd2a406b620 2023-11-24T06:08:53.586 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.580+0000 7fd2a0ff9700 1 -- 172.21.15.91:0/1130612962 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd29401c460 con 0x7fd2a406b620 2023-11-24T06:08:53.586 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.580+0000 7fd2a0ff9700 1 --2- 172.21.15.91:0/1130612962 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd28406aed0 0x7fd28406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.586 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.580+0000 7fd2a37fe700 1 --2- 172.21.15.91:0/1130612962 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd28406aed0 0x7fd28406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:53.587 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.580+0000 7fd2a0ff9700 1 -- 172.21.15.91:0/1130612962 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd294088e10 con 0x7fd2a406b620 2023-11-24T06:08:53.587 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.581+0000 7fd2a37fe700 1 --2- 172.21.15.91:0/1130612962 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd28406aed0 0x7fd28406d390 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd2a40c6420 tx=0x7fd28c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:53.588 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.585+0000 7fd2a0ff9700 1 -- 172.21.15.91:0/1130612962 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd2940542a0 con 0x7fd2a406b620 2023-11-24T06:08:53.855 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.852+0000 7fd2aa4a1700 1 -- 172.21.15.91:0/1130612962 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fd2a40c8c80 con 0x7fd2a406b620 2023-11-24T06:08:53.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.853+0000 7fd2a0ff9700 1 -- 172.21.15.91:0/1130612962 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7fd2940578c0 con 0x7fd2a406b620 2023-11-24T06:08:53.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.855+0000 7fd2aa4a1700 1 -- 172.21.15.91:0/1130612962 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd28406aed0 msgr2=0x7fd28406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:53.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.855+0000 7fd2aa4a1700 1 --2- 172.21.15.91:0/1130612962 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd28406aed0 0x7fd28406d390 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd2a40c6420 tx=0x7fd28c008040 comp rx=0 tx=0).stop 2023-11-24T06:08:53.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.855+0000 7fd2aa4a1700 1 -- 172.21.15.91:0/1130612962 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2a406b620 msgr2=0x7fd2a41f4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:53.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.855+0000 7fd2aa4a1700 1 --2- 172.21.15.91:0/1130612962 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2a406b620 0x7fd2a41f4720 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd29400c4d0 tx=0x7fd29400c890 comp rx=0 tx=0).stop 2023-11-24T06:08:53.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.856+0000 7fd2aa4a1700 1 -- 172.21.15.91:0/1130612962 shutdown_connections 2023-11-24T06:08:53.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.856+0000 7fd2aa4a1700 1 --2- 172.21.15.91:0/1130612962 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd28406aed0 0x7fd28406d390 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.856+0000 7fd2aa4a1700 1 --2- 172.21.15.91:0/1130612962 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd2a40b5ab0 0x7fd2a41f7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.856+0000 7fd2aa4a1700 1 --2- 172.21.15.91:0/1130612962 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2a406b620 0x7fd2a41f4720 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.856+0000 7fd2aa4a1700 1 --2- 172.21.15.91:0/1130612962 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd2a40b4150 0x7fd2a40cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.856+0000 7fd2aa4a1700 1 -- 172.21.15.91:0/1130612962 >> 172.21.15.91:0/1130612962 conn(0x7fd2a40b2740 msgr2=0x7fd2a40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:53.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.856+0000 7fd2aa4a1700 1 -- 172.21.15.91:0/1130612962 shutdown_connections 2023-11-24T06:08:53.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.856+0000 7fd2aa4a1700 1 -- 172.21.15.91:0/1130612962 wait complete. 2023-11-24T06:08:53.872 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-11-24T06:08:53.993 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.990+0000 7f7c1ed95700 1 Processor -- start 2023-11-24T06:08:53.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.990+0000 7f7c1ed95700 1 -- start start 2023-11-24T06:08:53.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.990+0000 7f7c1ed95700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f7c1815a440 0x7f7c180cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.990+0000 7f7c1ed95700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7c1815a820 0x7f7c180d04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.990+0000 7f7c1ed95700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c1815eb80 0x7f7c180726a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:53.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.990+0000 7f7c1ed95700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c180b05f0 con 0x7f7c1815a440 2023-11-24T06:08:53.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.990+0000 7f7c1ed95700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c1815f080 con 0x7f7c1815a820 2023-11-24T06:08:53.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.990+0000 7f7c1ed95700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c180afd50 con 0x7f7c1815eb80 2023-11-24T06:08:53.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.991+0000 7f7c1cb31700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f7c1815a440 0x7f7c180cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:53.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.991+0000 7f7c1cb31700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f7c1815a440 0x7f7c180cffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59490/0 (socket says 172.21.15.91:59490) 2023-11-24T06:08:53.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.991+0000 7f7c1cb31700 1 -- 172.21.15.91:0/4232578269 learned_addr learned my addr 172.21.15.91:0/4232578269 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:53.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.991+0000 7f7c1cb31700 1 -- 172.21.15.91:0/4232578269 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c1815eb80 msgr2=0x7f7c180726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:53.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.991+0000 7f7c1cb31700 1 --2- 172.21.15.91:0/4232578269 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c1815eb80 0x7f7c180726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.991+0000 7f7c1cb31700 1 -- 172.21.15.91:0/4232578269 >> v2:172.21.15.94:3300/0 conn(0x7f7c1815a820 msgr2=0x7f7c180d04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:53.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.991+0000 7f7c1cb31700 1 --2- 172.21.15.91:0/4232578269 >> v2:172.21.15.94:3300/0 conn(0x7f7c1815a820 0x7f7c180d04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.991+0000 7f7c1cb31700 1 -- 172.21.15.91:0/4232578269 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c180b0030 con 0x7f7c1815a440 2023-11-24T06:08:53.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.991+0000 7f7c1cb31700 1 --2- 172.21.15.91:0/4232578269 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7c1815a440 0x7f7c180cffa0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7c18069460 tx=0x7f7c0c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d22a9b4f8dcdfb8c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:53.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.991+0000 7f7c177fe700 1 -- 172.21.15.91:0/4232578269 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c00b870 con 0x7f7c1815a440 2023-11-24T06:08:53.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.991+0000 7f7c177fe700 1 -- 172.21.15.91:0/4232578269 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0c00b9d0 con 0x7f7c1815a440 2023-11-24T06:08:53.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.991+0000 7f7c177fe700 1 -- 172.21.15.91:0/4232578269 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c00bd60 con 0x7f7c1815a440 2023-11-24T06:08:53.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.992+0000 7f7c1ed95700 1 -- 172.21.15.91:0/4232578269 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7c1815a440 msgr2=0x7f7c180cffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:53.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.992+0000 7f7c1ed95700 1 --2- 172.21.15.91:0/4232578269 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7c1815a440 0x7f7c180cffa0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7c18069460 tx=0x7f7c0c009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.992+0000 7f7c1ed95700 1 -- 172.21.15.91:0/4232578269 shutdown_connections 2023-11-24T06:08:53.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.992+0000 7f7c1ed95700 1 --2- 172.21.15.91:0/4232578269 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c1815eb80 0x7f7c180726a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.992+0000 7f7c1ed95700 1 --2- 172.21.15.91:0/4232578269 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7c1815a440 0x7f7c180cffa0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.992+0000 7f7c1ed95700 1 --2- 172.21.15.91:0/4232578269 >> v2:172.21.15.94:3300/0 conn(0x7f7c1815a820 0x7f7c180d04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:53.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.992+0000 7f7c1ed95700 1 -- 172.21.15.91:0/4232578269 >> 172.21.15.91:0/4232578269 conn(0x7f7c180b5070 msgr2=0x7f7c180b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:53.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.992+0000 7f7c1ed95700 1 -- 172.21.15.91:0/4232578269 shutdown_connections 2023-11-24T06:08:53.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.992+0000 7f7c1ed95700 1 -- 172.21.15.91:0/4232578269 wait complete. 2023-11-24T06:08:53.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.993+0000 7f7c1ed95700 1 Processor -- start 2023-11-24T06:08:53.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.993+0000 7f7c1ed95700 1 -- start start 2023-11-24T06:08:54.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c1ed95700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7c1815a440 0x7f7c18144510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:54.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c1ed95700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c1815eb80 0x7f7c1815bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:54.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c1ed95700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c1806b930 0x7f7c180ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:54.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c1ed95700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c18144e50 con 0x7f7c1815a440 2023-11-24T06:08:54.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c1ed95700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1815c4d0 con 0x7f7c1806b930 2023-11-24T06:08:54.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c1ed95700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1815ac10 con 0x7f7c1815eb80 2023-11-24T06:08:54.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c1cb31700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7c1815a440 0x7f7c18144510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:54.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c1cb31700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7c1815a440 0x7f7c18144510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59496/0 (socket says 172.21.15.91:59496) 2023-11-24T06:08:54.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c1cb31700 1 -- 172.21.15.91:0/804967697 learned_addr learned my addr 172.21.15.91:0/804967697 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:54.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c17fff700 1 --2- 172.21.15.91:0/804967697 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c1815eb80 0x7f7c1815bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:54.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c1d332700 1 --2- 172.21.15.91:0/804967697 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c1806b930 0x7f7c180ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:54.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c1cb31700 1 -- 172.21.15.91:0/804967697 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c1815eb80 msgr2=0x7f7c1815bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:54.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c1cb31700 1 --2- 172.21.15.91:0/804967697 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c1815eb80 0x7f7c1815bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c1cb31700 1 -- 172.21.15.91:0/804967697 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c1806b930 msgr2=0x7f7c180ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:54.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c1cb31700 1 --2- 172.21.15.91:0/804967697 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c1806b930 0x7f7c180ceec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c1cb31700 1 -- 172.21.15.91:0/804967697 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c009550 con 0x7f7c1815a440 2023-11-24T06:08:54.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c1cb31700 1 --2- 172.21.15.91:0/804967697 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7c1815a440 0x7f7c18144510 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7c180737f0 tx=0x7f7c0c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:54.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c157fa700 1 -- 172.21.15.91:0/804967697 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c004310 con 0x7f7c1815a440 2023-11-24T06:08:54.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.994+0000 7f7c157fa700 1 -- 172.21.15.91:0/804967697 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0c01f070 con 0x7f7c1815a440 2023-11-24T06:08:54.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.995+0000 7f7c1ed95700 1 -- 172.21.15.91:0/804967697 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c1815aef0 con 0x7f7c1815a440 2023-11-24T06:08:54.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.995+0000 7f7c157fa700 1 -- 172.21.15.91:0/804967697 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c0036a0 con 0x7f7c1815a440 2023-11-24T06:08:54.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.995+0000 7f7c1ed95700 1 -- 172.21.15.91:0/804967697 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1815b4b0 con 0x7f7c1815a440 2023-11-24T06:08:54.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.995+0000 7f7c1ed95700 1 -- 172.21.15.91:0/804967697 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c18068670 con 0x7f7c1815a440 2023-11-24T06:08:54.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.996+0000 7f7c157fa700 1 -- 172.21.15.91:0/804967697 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7c0c021030 con 0x7f7c1815a440 2023-11-24T06:08:54.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.996+0000 7f7c157fa700 1 --2- 172.21.15.91:0/804967697 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7bfc06aed0 0x7f7bfc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:54.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.996+0000 7f7c157fa700 1 -- 172.21.15.91:0/804967697 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7c0c08a750 con 0x7f7c1815a440 2023-11-24T06:08:54.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.996+0000 7f7c17fff700 1 --2- 172.21.15.91:0/804967697 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7bfc06aed0 0x7f7bfc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:54.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:53.997+0000 7f7c17fff700 1 --2- 172.21.15.91:0/804967697 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7bfc06aed0 0x7f7bfc06d390 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7c04006fd0 tx=0x7f7c04008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:54.005 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.001+0000 7f7c157fa700 1 -- 172.21.15.91:0/804967697 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c0c055be0 con 0x7f7c1815a440 2023-11-24T06:08:54.273 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.270+0000 7f7c1ed95700 1 -- 172.21.15.91:0/804967697 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f7c180ae7b0 con 0x7f7c1815a440 2023-11-24T06:08:54.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.272+0000 7f7c157fa700 1 -- 172.21.15.91:0/804967697 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7c0c059200 con 0x7f7c1815a440 2023-11-24T06:08:54.278 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.274+0000 7f7c1ed95700 1 -- 172.21.15.91:0/804967697 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7bfc06aed0 msgr2=0x7f7bfc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:54.278 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.274+0000 7f7c1ed95700 1 --2- 172.21.15.91:0/804967697 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7bfc06aed0 0x7f7bfc06d390 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7c04006fd0 tx=0x7f7c04008040 comp rx=0 tx=0).stop 2023-11-24T06:08:54.278 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.274+0000 7f7c1ed95700 1 -- 172.21.15.91:0/804967697 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7c1815a440 msgr2=0x7f7c18144510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:54.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.274+0000 7f7c1ed95700 1 --2- 172.21.15.91:0/804967697 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7c1815a440 0x7f7c18144510 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7c180737f0 tx=0x7f7c0c004750 comp rx=0 tx=0).stop 2023-11-24T06:08:54.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.274+0000 7f7c1ed95700 1 -- 172.21.15.91:0/804967697 shutdown_connections 2023-11-24T06:08:54.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.274+0000 7f7c1ed95700 1 --2- 172.21.15.91:0/804967697 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7bfc06aed0 0x7f7bfc06d390 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.274+0000 7f7c1ed95700 1 --2- 172.21.15.91:0/804967697 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7c1806b930 0x7f7c180ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.274+0000 7f7c1ed95700 1 --2- 172.21.15.91:0/804967697 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7c1815a440 0x7f7c18144510 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.274+0000 7f7c1ed95700 1 --2- 172.21.15.91:0/804967697 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7c1815eb80 0x7f7c1815bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.274+0000 7f7c1ed95700 1 -- 172.21.15.91:0/804967697 >> 172.21.15.91:0/804967697 conn(0x7f7c180b5070 msgr2=0x7f7c180ad1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:54.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.274+0000 7f7c1ed95700 1 -- 172.21.15.91:0/804967697 shutdown_connections 2023-11-24T06:08:54.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.274+0000 7f7c1ed95700 1 -- 172.21.15.91:0/804967697 wait complete. 2023-11-24T06:08:54.290 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-11-24T06:08:54.411 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.408+0000 7f12c3735700 1 Processor -- start 2023-11-24T06:08:54.412 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.408+0000 7f12c3735700 1 -- start start 2023-11-24T06:08:54.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.409+0000 7f12c3735700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f12bc0b5ab0 0x7f12bc06e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:54.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.409+0000 7f12c3735700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12bc071f50 con 0x7f12bc0b4150 2023-11-24T06:08:54.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.409+0000 7f12c3735700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12bc072230 con 0x7f12bc06b620 2023-11-24T06:08:54.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.409+0000 7f12c3735700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12bc072510 con 0x7f12bc0b5ab0 2023-11-24T06:08:54.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.409+0000 7f12c1cd2700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f12bc0b5ab0 0x7f12bc06e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:54.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.409+0000 7f12c0cd0700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f12bc0b4150 0x7f12bc06de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54440/0 (socket says 172.21.15.91:54440) 2023-11-24T06:08:54.414 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.409+0000 7f12c0cd0700 1 -- 172.21.15.91:0/3187874979 learned_addr learned my addr 172.21.15.91:0/3187874979 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:54.414 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.409+0000 7f12c1cd2700 1 -- 172.21.15.91:0/3187874979 >> v1:172.21.15.94:6789/0 conn(0x7f12bc06b620 legacy=0x7f12bc0b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:54.414 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.409+0000 7f12c1cd2700 1 -- 172.21.15.91:0/3187874979 >> v1:172.21.15.91:6789/0 conn(0x7f12bc0b4150 legacy=0x7f12bc06de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:54.414 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.409+0000 7f12c1cd2700 1 -- 172.21.15.91:0/3187874979 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12bc0727f0 con 0x7f12bc0b5ab0 2023-11-24T06:08:54.414 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.409+0000 7f12b3fff700 1 -- 172.21.15.91:0/3187874979 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825013854 0 0) 0x7f12bc071f50 con 0x7f12bc0b4150 2023-11-24T06:08:54.414 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.409+0000 7f12c1cd2700 1 --2- 172.21.15.91:0/3187874979 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f12bc0b5ab0 0x7f12bc06e560 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009d90 tx=0x7f12b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31d22e22f48803e server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:54.414 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.409+0000 7f12b3fff700 1 -- 172.21.15.91:0/3187874979 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b400e070 con 0x7f12bc0b5ab0 2023-11-24T06:08:54.414 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.409+0000 7f12b3fff700 1 -- 172.21.15.91:0/3187874979 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b400bb90 con 0x7f12bc0b5ab0 2023-11-24T06:08:54.414 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.409+0000 7f12b3fff700 1 -- 172.21.15.91:0/3187874979 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b4003a20 con 0x7f12bc0b5ab0 2023-11-24T06:08:54.417 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.410+0000 7f12c3735700 1 -- 172.21.15.91:0/3187874979 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f12bc0b5ab0 msgr2=0x7f12bc06e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:54.417 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.410+0000 7f12c3735700 1 --2- 172.21.15.91:0/3187874979 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f12bc0b5ab0 0x7f12bc06e560 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f12b4009d90 tx=0x7f12b400b330 comp rx=0 tx=0).stop 2023-11-24T06:08:54.417 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.410+0000 7f12c3735700 1 -- 172.21.15.91:0/3187874979 shutdown_connections 2023-11-24T06:08:54.417 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.410+0000 7f12c3735700 1 --2- 172.21.15.91:0/3187874979 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f12bc0b5ab0 0x7f12bc06e560 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.417 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.410+0000 7f12c3735700 1 -- 172.21.15.91:0/3187874979 >> 172.21.15.91:0/3187874979 conn(0x7f12bc0b2740 msgr2=0x7f12bc0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:54.417 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.410+0000 7f12c3735700 1 -- 172.21.15.91:0/3187874979 shutdown_connections 2023-11-24T06:08:54.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.410+0000 7f12c3735700 1 -- 172.21.15.91:0/3187874979 wait complete. 2023-11-24T06:08:54.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.411+0000 7f12c3735700 1 Processor -- start 2023-11-24T06:08:54.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.411+0000 7f12c3735700 1 -- start start 2023-11-24T06:08:54.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c3735700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f12bc06b620 0x7f12bc142000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:54.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c3735700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12bc0b4150 0x7f12bc1e5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:54.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c3735700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f12bc0b5ab0 0x7f12bc1f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:54.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c3735700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc1f81d0 con 0x7f12bc06b620 2023-11-24T06:08:54.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c3735700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc0c7760 con 0x7f12bc0b4150 2023-11-24T06:08:54.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c3735700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc0c7a40 con 0x7f12bc0b5ab0 2023-11-24T06:08:54.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c14d1700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f12bc06b620 0x7f12bc142000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:54.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c14d1700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f12bc06b620 0x7f12bc142000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59510/0 (socket says 172.21.15.91:59510) 2023-11-24T06:08:54.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c14d1700 1 -- 172.21.15.91:0/672422749 learned_addr learned my addr 172.21.15.91:0/672422749 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:54.419 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c0cd0700 1 --2- 172.21.15.91:0/672422749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12bc0b4150 0x7f12bc1e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:54.419 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c1cd2700 1 --2- 172.21.15.91:0/672422749 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f12bc0b5ab0 0x7f12bc1f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:54.419 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c14d1700 1 -- 172.21.15.91:0/672422749 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f12bc0b5ab0 msgr2=0x7f12bc1f3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:54.419 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c14d1700 1 --2- 172.21.15.91:0/672422749 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f12bc0b5ab0 0x7f12bc1f3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.419 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c14d1700 1 -- 172.21.15.91:0/672422749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12bc0b4150 msgr2=0x7f12bc1e5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:54.419 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c14d1700 1 --2- 172.21.15.91:0/672422749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12bc0b4150 0x7f12bc1e5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.419 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c14d1700 1 -- 172.21.15.91:0/672422749 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b4009a70 con 0x7f12bc06b620 2023-11-24T06:08:54.419 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c1cd2700 1 --2- 172.21.15.91:0/672422749 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f12bc0b5ab0 0x7f12bc1f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:54.419 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.412+0000 7f12c14d1700 1 --2- 172.21.15.91:0/672422749 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f12bc06b620 0x7f12bc142000 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f12b800b4a0 tx=0x7f12b800b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:54.419 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.413+0000 7f12b1ffb700 1 -- 172.21.15.91:0/672422749 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b8011830 con 0x7f12bc06b620 2023-11-24T06:08:54.419 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.413+0000 7f12b1ffb700 1 -- 172.21.15.91:0/672422749 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b8011990 con 0x7f12bc06b620 2023-11-24T06:08:54.419 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.413+0000 7f12c3735700 1 -- 172.21.15.91:0/672422749 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12bc0c7d20 con 0x7f12bc06b620 2023-11-24T06:08:54.420 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.413+0000 7f12b1ffb700 1 -- 172.21.15.91:0/672422749 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b8011d40 con 0x7f12bc06b620 2023-11-24T06:08:54.420 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.413+0000 7f12c3735700 1 -- 172.21.15.91:0/672422749 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12bc1f4b90 con 0x7f12bc06b620 2023-11-24T06:08:54.421 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.414+0000 7f12c3735700 1 -- 172.21.15.91:0/672422749 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12bc0c8370 con 0x7f12bc06b620 2023-11-24T06:08:54.421 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.414+0000 7f12b1ffb700 1 -- 172.21.15.91:0/672422749 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f12b801a400 con 0x7f12bc06b620 2023-11-24T06:08:54.421 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.414+0000 7f12b1ffb700 1 --2- 172.21.15.91:0/672422749 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f12a006aed0 0x7f12a006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:54.421 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.414+0000 7f12b1ffb700 1 -- 172.21.15.91:0/672422749 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f12b8088ed0 con 0x7f12bc06b620 2023-11-24T06:08:54.421 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.415+0000 7f12c0cd0700 1 --2- 172.21.15.91:0/672422749 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f12a006aed0 0x7f12a006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:54.421 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.415+0000 7f12c0cd0700 1 --2- 172.21.15.91:0/672422749 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f12a006aed0 0x7f12a006d390 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f12ac007b10 tx=0x7f12ac007a60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:54.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.419+0000 7f12b1ffb700 1 -- 172.21.15.91:0/672422749 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f12b8054360 con 0x7f12bc06b620 2023-11-24T06:08:54.688 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.685+0000 7f12c3735700 1 -- 172.21.15.91:0/672422749 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f12bc068670 con 0x7f12bc06b620 2023-11-24T06:08:54.690 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.687+0000 7f12b1ffb700 1 -- 172.21.15.91:0/672422749 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f12b8057980 con 0x7f12bc06b620 2023-11-24T06:08:54.693 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.689+0000 7f12c3735700 1 -- 172.21.15.91:0/672422749 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f12a006aed0 msgr2=0x7f12a006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:54.693 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.689+0000 7f12c3735700 1 --2- 172.21.15.91:0/672422749 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f12a006aed0 0x7f12a006d390 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f12ac007b10 tx=0x7f12ac007a60 comp rx=0 tx=0).stop 2023-11-24T06:08:54.693 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.689+0000 7f12c3735700 1 -- 172.21.15.91:0/672422749 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f12bc06b620 msgr2=0x7f12bc142000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:54.693 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.689+0000 7f12c3735700 1 --2- 172.21.15.91:0/672422749 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f12bc06b620 0x7f12bc142000 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f12b800b4a0 tx=0x7f12b800b7b0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.693 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.689+0000 7f12c3735700 1 -- 172.21.15.91:0/672422749 shutdown_connections 2023-11-24T06:08:54.694 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.689+0000 7f12c3735700 1 --2- 172.21.15.91:0/672422749 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f12a006aed0 0x7f12a006d390 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.694 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.689+0000 7f12c3735700 1 --2- 172.21.15.91:0/672422749 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f12bc0b5ab0 0x7f12bc1f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.694 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.689+0000 7f12c3735700 1 --2- 172.21.15.91:0/672422749 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f12bc06b620 0x7f12bc142000 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.694 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.689+0000 7f12c3735700 1 --2- 172.21.15.91:0/672422749 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f12bc0b4150 0x7f12bc1e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.694 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.689+0000 7f12c3735700 1 -- 172.21.15.91:0/672422749 >> 172.21.15.91:0/672422749 conn(0x7f12bc0b2740 msgr2=0x7f12bc0afe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:54.694 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.689+0000 7f12c3735700 1 -- 172.21.15.91:0/672422749 shutdown_connections 2023-11-24T06:08:54.694 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.689+0000 7f12c3735700 1 -- 172.21.15.91:0/672422749 wait complete. 2023-11-24T06:08:54.705 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:54.825 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.821+0000 7fd20ebf2700 1 Processor -- start 2023-11-24T06:08:54.825 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.821+0000 7fd20ebf2700 1 -- start start 2023-11-24T06:08:54.828 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.822+0000 7fd20ebf2700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd20806ec70 0x7fd20806f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:54.828 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.822+0000 7fd20ebf2700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd20806f7c0 0x7fd2080ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:54.828 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.822+0000 7fd20ebf2700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd208144c90 con 0x7fd2081448b0 2023-11-24T06:08:54.828 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.822+0000 7fd20ebf2700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd20806fc50 con 0x7fd20806ec70 2023-11-24T06:08:54.828 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.822+0000 7fd20ebf2700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2080b4c80 con 0x7fd20806f7c0 2023-11-24T06:08:54.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.823+0000 7fd207fff700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd20806ec70 0x7fd20806f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:54.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.823+0000 7fd20d18f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd20806f7c0 0x7fd2080ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:54.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.823+0000 7fd20d18f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd20806f7c0 0x7fd2080ce2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:42954/0 (socket says 172.21.15.91:42954) 2023-11-24T06:08:54.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.823+0000 7fd20d18f700 1 -- 172.21.15.91:0/349331541 learned_addr learned my addr 172.21.15.91:0/349331541 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:54.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.823+0000 7fd207fff700 1 -- 172.21.15.91:0/349331541 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd20806f7c0 msgr2=0x7fd2080ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:54.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.823+0000 7fd207fff700 1 --2- 172.21.15.91:0/349331541 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd20806f7c0 0x7fd2080ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.823+0000 7fd207fff700 1 -- 172.21.15.91:0/349331541 >> v1:172.21.15.94:6789/0 conn(0x7fd2081448b0 legacy=0x7fd2080b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:54.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.823+0000 7fd207fff700 1 -- 172.21.15.91:0/349331541 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2080b4f60 con 0x7fd20806ec70 2023-11-24T06:08:54.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.823+0000 7fd20d18f700 1 --2- 172.21.15.91:0/349331541 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd20806f7c0 0x7fd2080ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:08:54.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.823+0000 7fd207fff700 1 --2- 172.21.15.91:0/349331541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20806ec70 0x7fd20806f280 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4009840 tx=0x7fd1f4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bef481b65134588 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:54.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.823+0000 7fd2077fe700 1 -- 172.21.15.91:0/349331541 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4011830 con 0x7fd20806ec70 2023-11-24T06:08:54.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.823+0000 7fd2077fe700 1 -- 172.21.15.91:0/349331541 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f4011990 con 0x7fd20806ec70 2023-11-24T06:08:54.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.823+0000 7fd2077fe700 1 -- 172.21.15.91:0/349331541 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4011d20 con 0x7fd20806ec70 2023-11-24T06:08:54.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.824+0000 7fd20ebf2700 1 -- 172.21.15.91:0/349331541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20806ec70 msgr2=0x7fd20806f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:54.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.824+0000 7fd20ebf2700 1 --2- 172.21.15.91:0/349331541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20806ec70 0x7fd20806f280 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4009840 tx=0x7fd1f4009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:54.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.824+0000 7fd20ebf2700 1 -- 172.21.15.91:0/349331541 shutdown_connections 2023-11-24T06:08:54.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.824+0000 7fd20ebf2700 1 --2- 172.21.15.91:0/349331541 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd20806f7c0 0x7fd2080ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.824+0000 7fd20ebf2700 1 --2- 172.21.15.91:0/349331541 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20806ec70 0x7fd20806f280 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.824+0000 7fd20ebf2700 1 -- 172.21.15.91:0/349331541 >> 172.21.15.91:0/349331541 conn(0x7fd2080d0bc0 msgr2=0x7fd2080d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:54.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.824+0000 7fd20ebf2700 1 -- 172.21.15.91:0/349331541 shutdown_connections 2023-11-24T06:08:54.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.824+0000 7fd20ebf2700 1 -- 172.21.15.91:0/349331541 wait complete. 2023-11-24T06:08:54.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.825+0000 7fd20ebf2700 1 Processor -- start 2023-11-24T06:08:54.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.826+0000 7fd20ebf2700 1 -- start start 2023-11-24T06:08:54.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20ebf2700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20806ec70 0x7fd208139d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:54.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20ebf2700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd20806f7c0 0x7fd2080ced50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:54.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20ebf2700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2081448b0 0x7fd208137100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:54.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20ebf2700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20813bd20 con 0x7fd2081448b0 2023-11-24T06:08:54.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20ebf2700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd208138e60 con 0x7fd20806ec70 2023-11-24T06:08:54.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20ebf2700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd208137f00 con 0x7fd20806f7c0 2023-11-24T06:08:54.835 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20d18f700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2081448b0 0x7fd208137100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:54.835 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20d18f700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2081448b0 0x7fd208137100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59528/0 (socket says 172.21.15.91:59528) 2023-11-24T06:08:54.835 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20d18f700 1 -- 172.21.15.91:0/1695963542 learned_addr learned my addr 172.21.15.91:0/1695963542 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:54.835 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20c98e700 1 --2- 172.21.15.91:0/1695963542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20806ec70 0x7fd208139d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:54.835 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd207fff700 1 --2- 172.21.15.91:0/1695963542 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd20806f7c0 0x7fd2080ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:54.835 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20d18f700 1 -- 172.21.15.91:0/1695963542 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd20806f7c0 msgr2=0x7fd2080ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:54.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20d18f700 1 --2- 172.21.15.91:0/1695963542 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd20806f7c0 0x7fd2080ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20d18f700 1 -- 172.21.15.91:0/1695963542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20806ec70 msgr2=0x7fd208139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:54.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20d18f700 1 --2- 172.21.15.91:0/1695963542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20806ec70 0x7fd208139d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:54.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20d18f700 1 -- 172.21.15.91:0/1695963542 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd200009b30 con 0x7fd2081448b0 2023-11-24T06:08:54.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20d18f700 1 --2- 172.21.15.91:0/1695963542 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2081448b0 0x7fd208137100 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd20000efc0 tx=0x7fd20000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:54.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd2057fa700 1 -- 172.21.15.91:0/1695963542 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd200003c20 con 0x7fd2081448b0 2023-11-24T06:08:54.836 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd2057fa700 1 -- 172.21.15.91:0/1695963542 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20000f040 con 0x7fd2081448b0 2023-11-24T06:08:54.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20ebf2700 1 -- 172.21.15.91:0/1695963542 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f40094a0 con 0x7fd2081448b0 2023-11-24T06:08:54.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd2057fa700 1 -- 172.21.15.91:0/1695963542 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2000092e0 con 0x7fd2081448b0 2023-11-24T06:08:54.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.827+0000 7fd20ebf2700 1 -- 172.21.15.91:0/1695963542 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2081387a0 con 0x7fd2081448b0 2023-11-24T06:08:54.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.828+0000 7fd20ebf2700 1 -- 172.21.15.91:0/1695963542 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd208068670 con 0x7fd2081448b0 2023-11-24T06:08:54.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.829+0000 7fd2057fa700 1 -- 172.21.15.91:0/1695963542 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd20000c880 con 0x7fd2081448b0 2023-11-24T06:08:54.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.829+0000 7fd2057fa700 1 --2- 172.21.15.91:0/1695963542 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd1ec06aed0 0x7fd1ec06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:54.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.829+0000 7fd2057fa700 1 -- 172.21.15.91:0/1695963542 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd200016070 con 0x7fd2081448b0 2023-11-24T06:08:54.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.829+0000 7fd20c98e700 1 --2- 172.21.15.91:0/1695963542 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd1ec06aed0 0x7fd1ec06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:54.837 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.830+0000 7fd20c98e700 1 --2- 172.21.15.91:0/1695963542 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd1ec06aed0 0x7fd1ec06d390 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc007fd0 tx=0x7fd1fc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:54.838 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:54.834+0000 7fd2057fa700 1 -- 172.21.15.91:0/1695963542 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd2000569c0 con 0x7fd2081448b0 2023-11-24T06:08:55.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.021+0000 7fd20ebf2700 1 -- 172.21.15.91:0/1695963542 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd208138fa0 con 0x7fd2081448b0 2023-11-24T06:08:55.026 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.022+0000 7fd2057fa700 1 -- 172.21.15.91:0/1695963542 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12773 (secure 0 0 0) 0x7fd208138fa0 con 0x7fd2081448b0 2023-11-24T06:08:55.026 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:55.026 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":30,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:51.739604+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:55.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.024+0000 7fd20ebf2700 1 -- 172.21.15.91:0/1695963542 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd1ec06aed0 msgr2=0x7fd1ec06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.024+0000 7fd20ebf2700 1 --2- 172.21.15.91:0/1695963542 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd1ec06aed0 0x7fd1ec06d390 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc007fd0 tx=0x7fd1fc009040 comp rx=0 tx=0).stop 2023-11-24T06:08:55.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.024+0000 7fd20ebf2700 1 -- 172.21.15.91:0/1695963542 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2081448b0 msgr2=0x7fd208137100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.024+0000 7fd20ebf2700 1 --2- 172.21.15.91:0/1695963542 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2081448b0 0x7fd208137100 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd20000efc0 tx=0x7fd20000c3a0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.024+0000 7fd20ebf2700 1 -- 172.21.15.91:0/1695963542 shutdown_connections 2023-11-24T06:08:55.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.024+0000 7fd20ebf2700 1 --2- 172.21.15.91:0/1695963542 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd1ec06aed0 0x7fd1ec06d390 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.024+0000 7fd20ebf2700 1 --2- 172.21.15.91:0/1695963542 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd20806ec70 0x7fd208139d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.024+0000 7fd20ebf2700 1 --2- 172.21.15.91:0/1695963542 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd2081448b0 0x7fd208137100 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.024+0000 7fd20ebf2700 1 --2- 172.21.15.91:0/1695963542 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd20806f7c0 0x7fd2080ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.024+0000 7fd20ebf2700 1 -- 172.21.15.91:0/1695963542 >> 172.21.15.91:0/1695963542 conn(0x7fd2080d0bc0 msgr2=0x7fd2080b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:55.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.024+0000 7fd20ebf2700 1 -- 172.21.15.91:0/1695963542 shutdown_connections 2023-11-24T06:08:55.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.024+0000 7fd20ebf2700 1 -- 172.21.15.91:0/1695963542 wait complete. 2023-11-24T06:08:55.041 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:08:55.042 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:08:55.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.161+0000 7fcb006e1700 1 Processor -- start 2023-11-24T06:08:55.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.161+0000 7fcb006e1700 1 -- start start 2023-11-24T06:08:55.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcb006e1700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fcaf815bde0 0x7fcaf815c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcb006e1700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcaf80cf2b0 0x7fcaf80d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcb006e1700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf80d0a90 con 0x7fcaf80ce5c0 2023-11-24T06:08:55.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcb006e1700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf806ee10 con 0x7fcaf815bde0 2023-11-24T06:08:55.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcb006e1700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf806f0f0 con 0x7fcaf80cf2b0 2023-11-24T06:08:55.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcafe47d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fcaf815bde0 0x7fcaf815c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcafec7e700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fcaf80ce5c0 0x7fcaf80cc670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54470/0 (socket says 172.21.15.91:54470) 2023-11-24T06:08:55.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcafec7e700 1 -- 172.21.15.91:0/262153340 learned_addr learned my addr 172.21.15.91:0/262153340 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:55.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcafdc7c700 1 --2- 172.21.15.91:0/262153340 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcaf80cf2b0 0x7fcaf80d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcafe47d700 1 -- 172.21.15.91:0/262153340 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcaf80cf2b0 msgr2=0x7fcaf80d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:55.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcafe47d700 1 --2- 172.21.15.91:0/262153340 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcaf80cf2b0 0x7fcaf80d01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcafe47d700 1 -- 172.21.15.91:0/262153340 >> v1:172.21.15.91:6789/0 conn(0x7fcaf80ce5c0 legacy=0x7fcaf80cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcafe47d700 1 -- 172.21.15.91:0/262153340 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf806f3d0 con 0x7fcaf815bde0 2023-11-24T06:08:55.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcafd47b700 1 -- 172.21.15.91:0/262153340 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 313490680 0 0) 0x7fcaf80d0a90 con 0x7fcaf80ce5c0 2023-11-24T06:08:55.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcafe47d700 1 --2- 172.21.15.91:0/262153340 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcaf815bde0 0x7fcaf815c1c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009840 tx=0x7fcaf4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b9d7dfdf32c103 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:55.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcafd47b700 1 -- 172.21.15.91:0/262153340 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4011830 con 0x7fcaf815bde0 2023-11-24T06:08:55.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcafd47b700 1 -- 172.21.15.91:0/262153340 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf4011990 con 0x7fcaf815bde0 2023-11-24T06:08:55.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.162+0000 7fcafd47b700 1 -- 172.21.15.91:0/262153340 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4011d20 con 0x7fcaf815bde0 2023-11-24T06:08:55.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.163+0000 7fcb006e1700 1 -- 172.21.15.91:0/262153340 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcaf815bde0 msgr2=0x7fcaf815c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.163+0000 7fcb006e1700 1 --2- 172.21.15.91:0/262153340 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcaf815bde0 0x7fcaf815c1c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009840 tx=0x7fcaf4009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:55.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.163+0000 7fcb006e1700 1 -- 172.21.15.91:0/262153340 shutdown_connections 2023-11-24T06:08:55.169 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.163+0000 7fcb006e1700 1 --2- 172.21.15.91:0/262153340 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcaf815bde0 0x7fcaf815c1c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.169 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.163+0000 7fcb006e1700 1 --2- 172.21.15.91:0/262153340 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcaf80cf2b0 0x7fcaf80d01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.169 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.163+0000 7fcb006e1700 1 -- 172.21.15.91:0/262153340 >> 172.21.15.91:0/262153340 conn(0x7fcaf80b3720 msgr2=0x7fcaf80b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:55.169 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.163+0000 7fcb006e1700 1 -- 172.21.15.91:0/262153340 shutdown_connections 2023-11-24T06:08:55.169 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.163+0000 7fcb006e1700 1 -- 172.21.15.91:0/262153340 wait complete. 2023-11-24T06:08:55.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.164+0000 7fcb006e1700 1 Processor -- start 2023-11-24T06:08:55.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.165+0000 7fcb006e1700 1 -- start start 2023-11-24T06:08:55.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.165+0000 7fcb006e1700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcaf80ce5c0 0x7fcaf81593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.165+0000 7fcb006e1700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcaf80cf2b0 0x7fcaf80c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.165+0000 7fcb006e1700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcaf815bde0 0x7fcaf80b4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.165+0000 7fcb006e1700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf815efe0 con 0x7fcaf815bde0 2023-11-24T06:08:55.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.165+0000 7fcb006e1700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf815c260 con 0x7fcaf80cf2b0 2023-11-24T06:08:55.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.165+0000 7fcb006e1700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf80b5310 con 0x7fcaf80ce5c0 2023-11-24T06:08:55.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.166+0000 7fcafec7e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcaf815bde0 0x7fcaf80b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.166+0000 7fcafec7e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcaf815bde0 0x7fcaf80b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59542/0 (socket says 172.21.15.91:59542) 2023-11-24T06:08:55.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.166+0000 7fcafec7e700 1 -- 172.21.15.91:0/2533169074 learned_addr learned my addr 172.21.15.91:0/2533169074 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:55.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.166+0000 7fcafe47d700 1 --2- 172.21.15.91:0/2533169074 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcaf80ce5c0 0x7fcaf81593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.166+0000 7fcafdc7c700 1 --2- 172.21.15.91:0/2533169074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcaf80cf2b0 0x7fcaf80c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.166+0000 7fcafec7e700 1 -- 172.21.15.91:0/2533169074 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcaf80ce5c0 msgr2=0x7fcaf81593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.166+0000 7fcafec7e700 1 --2- 172.21.15.91:0/2533169074 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcaf80ce5c0 0x7fcaf81593c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.166+0000 7fcafec7e700 1 -- 172.21.15.91:0/2533169074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcaf80cf2b0 msgr2=0x7fcaf80c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.166+0000 7fcafec7e700 1 --2- 172.21.15.91:0/2533169074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcaf80cf2b0 0x7fcaf80c90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.166+0000 7fcafec7e700 1 -- 172.21.15.91:0/2533169074 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf40094a0 con 0x7fcaf815bde0 2023-11-24T06:08:55.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.166+0000 7fcafec7e700 1 --2- 172.21.15.91:0/2533169074 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcaf815bde0 0x7fcaf80b4810 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0002230 tx=0x7fcaf000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:55.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.166+0000 7fcaeeffd700 1 -- 172.21.15.91:0/2533169074 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf000ac20 con 0x7fcaf815bde0 2023-11-24T06:08:55.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.166+0000 7fcaeeffd700 1 -- 172.21.15.91:0/2533169074 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf000ada0 con 0x7fcaf815bde0 2023-11-24T06:08:55.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.166+0000 7fcb006e1700 1 -- 172.21.15.91:0/2533169074 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf80b55f0 con 0x7fcaf815bde0 2023-11-24T06:08:55.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.166+0000 7fcaeeffd700 1 -- 172.21.15.91:0/2533169074 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf0007310 con 0x7fcaf815bde0 2023-11-24T06:08:55.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.166+0000 7fcb006e1700 1 -- 172.21.15.91:0/2533169074 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf815c5b0 con 0x7fcaf815bde0 2023-11-24T06:08:55.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.167+0000 7fcb006e1700 1 -- 172.21.15.91:0/2533169074 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf80b6780 con 0x7fcaf815bde0 2023-11-24T06:08:55.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.168+0000 7fcaeeffd700 1 -- 172.21.15.91:0/2533169074 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcaf0007470 con 0x7fcaf815bde0 2023-11-24T06:08:55.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.168+0000 7fcaeeffd700 1 --2- 172.21.15.91:0/2533169074 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcadc06aed0 0x7fcadc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.168+0000 7fcaeeffd700 1 -- 172.21.15.91:0/2533169074 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcaf00890e0 con 0x7fcaf815bde0 2023-11-24T06:08:55.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.168+0000 7fcafe47d700 1 --2- 172.21.15.91:0/2533169074 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcadc06aed0 0x7fcadc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.168+0000 7fcafe47d700 1 --2- 172.21.15.91:0/2533169074 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcadc06aed0 0x7fcadc06d390 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009b90 tx=0x7fcaf4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:55.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.172+0000 7fcaeeffd700 1 -- 172.21.15.91:0/2533169074 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcaf0054570 con 0x7fcaf815bde0 2023-11-24T06:08:55.364 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.360+0000 7fcb006e1700 1 -- 172.21.15.91:0/2533169074 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcaf8075480 con 0x7fcaf815bde0 2023-11-24T06:08:55.364 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.360+0000 7fcaeeffd700 1 -- 172.21.15.91:0/2533169074 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcaf0018020 con 0x7fcaf815bde0 2023-11-24T06:08:55.365 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:08:55.367 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.362+0000 7fcb006e1700 1 -- 172.21.15.91:0/2533169074 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcadc06aed0 msgr2=0x7fcadc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.367 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.362+0000 7fcb006e1700 1 --2- 172.21.15.91:0/2533169074 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcadc06aed0 0x7fcadc06d390 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009b90 tx=0x7fcaf4008030 comp rx=0 tx=0).stop 2023-11-24T06:08:55.367 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.362+0000 7fcb006e1700 1 -- 172.21.15.91:0/2533169074 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcaf815bde0 msgr2=0x7fcaf80b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.367 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.362+0000 7fcb006e1700 1 --2- 172.21.15.91:0/2533169074 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcaf815bde0 0x7fcaf80b4810 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0002230 tx=0x7fcaf000a060 comp rx=0 tx=0).stop 2023-11-24T06:08:55.367 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.362+0000 7fcb006e1700 1 -- 172.21.15.91:0/2533169074 shutdown_connections 2023-11-24T06:08:55.367 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.362+0000 7fcb006e1700 1 --2- 172.21.15.91:0/2533169074 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcadc06aed0 0x7fcadc06d390 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.367 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.362+0000 7fcb006e1700 1 --2- 172.21.15.91:0/2533169074 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcaf815bde0 0x7fcaf80b4810 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.367 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.362+0000 7fcb006e1700 1 --2- 172.21.15.91:0/2533169074 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcaf80ce5c0 0x7fcaf81593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.367 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.362+0000 7fcb006e1700 1 --2- 172.21.15.91:0/2533169074 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcaf80cf2b0 0x7fcaf80c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.362+0000 7fcb006e1700 1 -- 172.21.15.91:0/2533169074 >> 172.21.15.91:0/2533169074 conn(0x7fcaf80b3720 msgr2=0x7fcaf80b01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:55.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.362+0000 7fcb006e1700 1 -- 172.21.15.91:0/2533169074 shutdown_connections 2023-11-24T06:08:55.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.363+0000 7fcb006e1700 1 -- 172.21.15.91:0/2533169074 wait complete. 2023-11-24T06:08:55.378 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-11-24T06:08:55.499 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.495+0000 7f1608f3c700 1 Processor -- start 2023-11-24T06:08:55.499 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.495+0000 7f1608f3c700 1 -- start start 2023-11-24T06:08:55.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.496+0000 7f1608f3c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1604143ae0 0x7f16040b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.496+0000 7f1608f3c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f16040722c0 0x7f160406e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.496+0000 7f1608f3c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1604143ec0 con 0x7f16041440f0 2023-11-24T06:08:55.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.496+0000 7f1608f3c700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16040b40a0 con 0x7f16040722c0 2023-11-24T06:08:55.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.496+0000 7f1608f3c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16040b3ae0 con 0x7f1604143ae0 2023-11-24T06:08:55.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.496+0000 7f1602d9d700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f16040722c0 0x7f160406e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.496+0000 7f160259c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1604143ae0 0x7f16040b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.496+0000 7f160259c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1604143ae0 0x7f16040b51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:42978/0 (socket says 172.21.15.91:42978) 2023-11-24T06:08:55.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.496+0000 7f1602d9d700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f16040722c0 0x7f160406e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:43942/0 (socket says 172.21.15.91:43942) 2023-11-24T06:08:55.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.496+0000 7f160259c700 1 -- 172.21.15.91:0/3536056491 learned_addr learned my addr 172.21.15.91:0/3536056491 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:55.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.496+0000 7f160259c700 1 -- 172.21.15.91:0/3536056491 >> v2:172.21.15.94:3300/0 conn(0x7f16040722c0 msgr2=0x7f160406e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:55.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.496+0000 7f160259c700 1 --2- 172.21.15.91:0/3536056491 >> v2:172.21.15.94:3300/0 conn(0x7f16040722c0 0x7f160406e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.496+0000 7f160259c700 1 -- 172.21.15.91:0/3536056491 >> v1:172.21.15.94:6789/0 conn(0x7f16041440f0 legacy=0x7f1604071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.496+0000 7f160259c700 1 -- 172.21.15.91:0/3536056491 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16040b3dc0 con 0x7f1604143ae0 2023-11-24T06:08:55.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.497+0000 7f160259c700 1 --2- 172.21.15.91:0/3536056491 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1604143ae0 0x7f16040b51b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f15f40098f0 tx=0x7f15f4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e715a1bade06ee94 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:55.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.497+0000 7f160159a700 1 -- 172.21.15.91:0/3536056491 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f4011830 con 0x7f1604143ae0 2023-11-24T06:08:55.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.497+0000 7f160159a700 1 -- 172.21.15.91:0/3536056491 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f4011990 con 0x7f1604143ae0 2023-11-24T06:08:55.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.497+0000 7f160159a700 1 -- 172.21.15.91:0/3536056491 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f4011d20 con 0x7f1604143ae0 2023-11-24T06:08:55.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.497+0000 7f1608f3c700 1 -- 172.21.15.91:0/3536056491 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1604143ae0 msgr2=0x7f16040b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.497+0000 7f1608f3c700 1 --2- 172.21.15.91:0/3536056491 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1604143ae0 0x7f16040b51b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f15f40098f0 tx=0x7f15f4009c00 comp rx=0 tx=0).stop 2023-11-24T06:08:55.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.497+0000 7f1608f3c700 1 -- 172.21.15.91:0/3536056491 shutdown_connections 2023-11-24T06:08:55.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.497+0000 7f1608f3c700 1 --2- 172.21.15.91:0/3536056491 >> v2:172.21.15.94:3300/0 conn(0x7f16040722c0 0x7f160406e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.497+0000 7f1608f3c700 1 --2- 172.21.15.91:0/3536056491 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1604143ae0 0x7f16040b51b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.497+0000 7f1608f3c700 1 -- 172.21.15.91:0/3536056491 >> 172.21.15.91:0/3536056491 conn(0x7f16040b2740 msgr2=0x7f16040b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:55.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.497+0000 7f1608f3c700 1 -- 172.21.15.91:0/3536056491 shutdown_connections 2023-11-24T06:08:55.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.497+0000 7f1608f3c700 1 -- 172.21.15.91:0/3536056491 wait complete. 2023-11-24T06:08:55.506 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.499+0000 7f1608f3c700 1 Processor -- start 2023-11-24T06:08:55.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.499+0000 7f1608f3c700 1 -- start start 2023-11-24T06:08:55.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.500+0000 7f1608f3c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16040722c0 0x7f1604141e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.500+0000 7f1608f3c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1604143ae0 0x7f1604153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.500+0000 7f1608f3c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16041440f0 0x7f16041540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.500+0000 7f1608f3c700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1604153760 con 0x7f1604143ae0 2023-11-24T06:08:55.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.500+0000 7f1608f3c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1604152ae0 con 0x7f16041440f0 2023-11-24T06:08:55.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.500+0000 7f1608f3c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1604152dc0 con 0x7f16040722c0 2023-11-24T06:08:55.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.500+0000 7f1601d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1604143ae0 0x7f1604153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.500+0000 7f1601d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1604143ae0 0x7f1604153b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59554/0 (socket says 172.21.15.91:59554) 2023-11-24T06:08:55.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.500+0000 7f1601d9b700 1 -- 172.21.15.91:0/342464217 learned_addr learned my addr 172.21.15.91:0/342464217 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:55.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.500+0000 7f1602d9d700 1 --2- 172.21.15.91:0/342464217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16041440f0 0x7f16041540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.500+0000 7f160259c700 1 --2- 172.21.15.91:0/342464217 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16040722c0 0x7f1604141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.501+0000 7f1601d9b700 1 -- 172.21.15.91:0/342464217 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16040722c0 msgr2=0x7f1604141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.501+0000 7f1601d9b700 1 --2- 172.21.15.91:0/342464217 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16040722c0 0x7f1604141e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.501+0000 7f1601d9b700 1 -- 172.21.15.91:0/342464217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16041440f0 msgr2=0x7f16041540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.501+0000 7f1601d9b700 1 --2- 172.21.15.91:0/342464217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16041440f0 0x7f16041540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.501+0000 7f1601d9b700 1 -- 172.21.15.91:0/342464217 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f4009550 con 0x7f1604143ae0 2023-11-24T06:08:55.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.501+0000 7f1601d9b700 1 --2- 172.21.15.91:0/342464217 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1604143ae0 0x7f1604153b80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f15f800c450 tx=0x7f15f800c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:55.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.501+0000 7f15f2ffd700 1 -- 172.21.15.91:0/342464217 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f80052d0 con 0x7f1604143ae0 2023-11-24T06:08:55.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.501+0000 7f15f2ffd700 1 -- 172.21.15.91:0/342464217 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f8005430 con 0x7f1604143ae0 2023-11-24T06:08:55.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.501+0000 7f1608f3c700 1 -- 172.21.15.91:0/342464217 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16041530a0 con 0x7f1604143ae0 2023-11-24T06:08:55.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.501+0000 7f15f2ffd700 1 -- 172.21.15.91:0/342464217 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f80038c0 con 0x7f1604143ae0 2023-11-24T06:08:55.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.501+0000 7f1608f3c700 1 -- 172.21.15.91:0/342464217 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16041527b0 con 0x7f1604143ae0 2023-11-24T06:08:55.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.502+0000 7f1608f3c700 1 -- 172.21.15.91:0/342464217 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16041538a0 con 0x7f1604143ae0 2023-11-24T06:08:55.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.502+0000 7f15f2ffd700 1 -- 172.21.15.91:0/342464217 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f15f8003a20 con 0x7f1604143ae0 2023-11-24T06:08:55.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.503+0000 7f15f2ffd700 1 --2- 172.21.15.91:0/342464217 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15e406aed0 0x7f15e406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.503+0000 7f15f2ffd700 1 -- 172.21.15.91:0/342464217 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f15f8088510 con 0x7f1604143ae0 2023-11-24T06:08:55.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.503+0000 7f160259c700 1 --2- 172.21.15.91:0/342464217 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15e406aed0 0x7f15e406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.503+0000 7f160259c700 1 --2- 172.21.15.91:0/342464217 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15e406aed0 0x7f15e406d390 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f15f4009c40 tx=0x7f15f4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:55.511 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.507+0000 7f15f2ffd700 1 -- 172.21.15.91:0/342464217 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f16041538a0 con 0x7f1604143ae0 2023-11-24T06:08:55.701 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.697+0000 7f1608f3c700 1 -- 172.21.15.91:0/342464217 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f16041538a0 con 0x7f1604143ae0 2023-11-24T06:08:55.701 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.698+0000 7f15f2ffd700 1 -- 172.21.15.91:0/342464217 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7f15f8017060 con 0x7f1604143ae0 2023-11-24T06:08:55.702 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:55.702 INFO:teuthology.orchestra.run.smithi091.stdout:[ 2023-11-24T06:08:55.702 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:08:55.702 INFO:teuthology.orchestra.run.smithi091.stdout: "poolnum": 1, 2023-11-24T06:08:55.702 INFO:teuthology.orchestra.run.smithi091.stdout: "poolname": ".mgr" 2023-11-24T06:08:55.702 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:08:55.702 INFO:teuthology.orchestra.run.smithi091.stdout:] 2023-11-24T06:08:55.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.700+0000 7f1608f3c700 1 -- 172.21.15.91:0/342464217 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15e406aed0 msgr2=0x7f15e406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.700+0000 7f1608f3c700 1 --2- 172.21.15.91:0/342464217 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15e406aed0 0x7f15e406d390 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f15f4009c40 tx=0x7f15f4008030 comp rx=0 tx=0).stop 2023-11-24T06:08:55.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.700+0000 7f1608f3c700 1 -- 172.21.15.91:0/342464217 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1604143ae0 msgr2=0x7f1604153b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.700+0000 7f1608f3c700 1 --2- 172.21.15.91:0/342464217 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1604143ae0 0x7f1604153b80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f15f800c450 tx=0x7f15f800c760 comp rx=0 tx=0).stop 2023-11-24T06:08:55.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.700+0000 7f1608f3c700 1 -- 172.21.15.91:0/342464217 shutdown_connections 2023-11-24T06:08:55.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.700+0000 7f1608f3c700 1 --2- 172.21.15.91:0/342464217 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f15e406aed0 0x7f15e406d390 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.704 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.700+0000 7f1608f3c700 1 --2- 172.21.15.91:0/342464217 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f16041440f0 0x7f16041540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.705 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.700+0000 7f1608f3c700 1 --2- 172.21.15.91:0/342464217 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f16040722c0 0x7f1604141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.705 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.700+0000 7f1608f3c700 1 --2- 172.21.15.91:0/342464217 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1604143ae0 0x7f1604153b80 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.705 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.700+0000 7f1608f3c700 1 -- 172.21.15.91:0/342464217 >> 172.21.15.91:0/342464217 conn(0x7f16040b2740 msgr2=0x7f16040b41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:55.705 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.700+0000 7f1608f3c700 1 -- 172.21.15.91:0/342464217 shutdown_connections 2023-11-24T06:08:55.705 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.700+0000 7f1608f3c700 1 -- 172.21.15.91:0/342464217 wait complete. 2023-11-24T06:08:55.716 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-11-24T06:08:55.716 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-11-24T06:08:55.841 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.838+0000 7f2cdf22d700 1 Processor -- start 2023-11-24T06:08:55.842 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.838+0000 7f2cdf22d700 1 -- start start 2023-11-24T06:08:55.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cdf22d700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f2cd815bfe0 0x7f2cd80cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cdf22d700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd806f4b0 con 0x7f2cd80cefc0 2023-11-24T06:08:55.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cdf22d700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd815c3c0 con 0x7f2cd80ce430 2023-11-24T06:08:55.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cdf22d700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd80cd760 con 0x7f2cd815bfe0 2023-11-24T06:08:55.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cdd7ca700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f2cd80cefc0 0x7f2cd806f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54498/0 (socket says 172.21.15.91:54498) 2023-11-24T06:08:55.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cdd7ca700 1 -- 172.21.15.91:0/201862737 learned_addr learned my addr 172.21.15.91:0/201862737 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:55.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cd77fe700 1 -- 172.21.15.91:0/201862737 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78126481 0 0) 0x7f2cd806f4b0 con 0x7f2cd80cefc0 2023-11-24T06:08:55.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cd77fe700 1 -- 172.21.15.91:0/201862737 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc80035b0 con 0x7f2cd80cefc0 2023-11-24T06:08:55.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cdcfc9700 1 --2- 172.21.15.91:0/201862737 >> v2:172.21.15.94:3300/0 conn(0x7f2cd815bfe0 0x7f2cd80cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.845 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cd77fe700 1 -- 172.21.15.91:0/201862737 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2964877729 0 0) 0x7f2cc80035b0 con 0x7f2cd80cefc0 2023-11-24T06:08:55.846 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cd77fe700 1 -- 172.21.15.91:0/201862737 >> v2:172.21.15.94:3300/0 conn(0x7f2cd815bfe0 msgr2=0x7f2cd80cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:55.846 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cd77fe700 1 --2- 172.21.15.91:0/201862737 >> v2:172.21.15.94:3300/0 conn(0x7f2cd815bfe0 0x7f2cd80cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.846 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cd77fe700 1 -- 172.21.15.91:0/201862737 >> v1:172.21.15.94:6789/0 conn(0x7f2cd80ce430 legacy=0x7f2cd80ce8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.846 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cd77fe700 1 -- 172.21.15.91:0/201862737 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd806ee60 con 0x7f2cd80cefc0 2023-11-24T06:08:55.846 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cd77fe700 1 -- 172.21.15.91:0/201862737 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f2cd00019f0 con 0x7f2cd80cefc0 2023-11-24T06:08:55.846 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cd77fe700 1 -- 172.21.15.91:0/201862737 >> v1:172.21.15.91:6789/0 conn(0x7f2cd80cefc0 legacy=0x7f2cd806f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.846 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cd77fe700 1 --2- 172.21.15.91:0/201862737 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2cc8003a00 0x7f2cc8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.846 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cd77fe700 1 --2- 172.21.15.91:0/201862737 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cc8004850 0x7f2cc8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.846 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cd77fe700 1 --2- 172.21.15.91:0/201862737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cc8005210 0x7f2cc800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.846 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cd77fe700 1 -- 172.21.15.91:0/201862737 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd80cd760 con 0x7f2cc8003a00 2023-11-24T06:08:55.846 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cd77fe700 1 -- 172.21.15.91:0/201862737 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc800c0b0 con 0x7f2cc8005210 2023-11-24T06:08:55.846 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cd77fe700 1 -- 172.21.15.91:0/201862737 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc800c270 con 0x7f2cc8004850 2023-11-24T06:08:55.846 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cdcfc9700 1 --2- 172.21.15.91:0/201862737 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2cc8003a00 0x7f2cc8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.847 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cd7fff700 1 --2- 172.21.15.91:0/201862737 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cc8004850 0x7f2cc8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.847 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.839+0000 7f2cdd7ca700 1 --2- 172.21.15.91:0/201862737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cc8005210 0x7f2cc800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.847 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.840+0000 7f2cdcfc9700 1 -- 172.21.15.91:0/201862737 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cc8004850 msgr2=0x7f2cc8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:55.847 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.840+0000 7f2cdcfc9700 1 --2- 172.21.15.91:0/201862737 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cc8004850 0x7f2cc8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.847 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.840+0000 7f2cdcfc9700 1 -- 172.21.15.91:0/201862737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cc8005210 msgr2=0x7f2cc800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:55.847 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.840+0000 7f2cdcfc9700 1 --2- 172.21.15.91:0/201862737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cc8005210 0x7f2cc800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.847 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.840+0000 7f2cdcfc9700 1 -- 172.21.15.91:0/201862737 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc80035b0 con 0x7f2cc8003a00 2023-11-24T06:08:55.847 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.840+0000 7f2cdd7ca700 1 --2- 172.21.15.91:0/201862737 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cc8005210 0x7f2cc800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:55.847 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.840+0000 7f2cdd7ca700 1 -- 172.21.15.91:0/201862737 reap_dead start 2023-11-24T06:08:55.847 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.840+0000 7f2cdcfc9700 1 --2- 172.21.15.91:0/201862737 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2cc8003a00 0x7f2cc8003e20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc009b60 tx=0x7f2cc8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18c26dc1265e197c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:55.848 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.840+0000 7f2cd77fe700 1 -- 172.21.15.91:0/201862737 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc00e6c0 con 0x7f2cc8003a00 2023-11-24T06:08:55.848 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.840+0000 7f2cd77fe700 1 -- 172.21.15.91:0/201862737 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ccc00e820 con 0x7f2cc8003a00 2023-11-24T06:08:55.848 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.840+0000 7f2cd77fe700 1 -- 172.21.15.91:0/201862737 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc00eb90 con 0x7f2cc8003a00 2023-11-24T06:08:55.848 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.840+0000 7f2cdf22d700 1 -- 172.21.15.91:0/201862737 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2cc8003a00 msgr2=0x7f2cc8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.848 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.840+0000 7f2cdf22d700 1 --2- 172.21.15.91:0/201862737 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2cc8003a00 0x7f2cc8003e20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc009b60 tx=0x7f2cc8005880 comp rx=0 tx=0).stop 2023-11-24T06:08:55.848 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.840+0000 7f2cdf22d700 1 -- 172.21.15.91:0/201862737 shutdown_connections 2023-11-24T06:08:55.848 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.840+0000 7f2cdf22d700 1 --2- 172.21.15.91:0/201862737 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2cc8003a00 0x7f2cc8003e20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.848 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.840+0000 7f2cdf22d700 1 -- 172.21.15.91:0/201862737 >> 172.21.15.91:0/201862737 conn(0x7f2cd80b3700 msgr2=0x7f2cd80b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:55.848 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.841+0000 7f2cdf22d700 1 -- 172.21.15.91:0/201862737 shutdown_connections 2023-11-24T06:08:55.848 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.841+0000 7f2cdf22d700 1 -- 172.21.15.91:0/201862737 wait complete. 2023-11-24T06:08:55.851 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.842+0000 7f2cdf22d700 1 Processor -- start 2023-11-24T06:08:55.851 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.842+0000 7f2cdf22d700 1 -- start start 2023-11-24T06:08:55.851 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.843+0000 7f2cdf22d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2cc8003a00 0x7f2cd815a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.851 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.843+0000 7f2cdf22d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cd80c7160 0x7f2cd80c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.851 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.843+0000 7f2cdf22d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cd80caa40 0x7f2cd80caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.851 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.843+0000 7f2cdf22d700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd80cb410 con 0x7f2cc8003a00 2023-11-24T06:08:55.851 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.843+0000 7f2cdf22d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd80b4cc0 con 0x7f2cd80c7160 2023-11-24T06:08:55.851 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.843+0000 7f2cdf22d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd80c7640 con 0x7f2cd80caa40 2023-11-24T06:08:55.851 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.843+0000 7f2cdcfc9700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2cc8003a00 0x7f2cd815a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.843+0000 7f2cdcfc9700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2cc8003a00 0x7f2cd815a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59574/0 (socket says 172.21.15.91:59574) 2023-11-24T06:08:55.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.843+0000 7f2cdcfc9700 1 -- 172.21.15.91:0/1876907227 learned_addr learned my addr 172.21.15.91:0/1876907227 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:55.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.844+0000 7f2cd7fff700 1 --2- 172.21.15.91:0/1876907227 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cd80c7160 0x7f2cd80c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.844+0000 7f2cdd7ca700 1 --2- 172.21.15.91:0/1876907227 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cd80caa40 0x7f2cd80caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.844+0000 7f2cdcfc9700 1 -- 172.21.15.91:0/1876907227 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cd80caa40 msgr2=0x7f2cd80caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.844+0000 7f2cdcfc9700 1 --2- 172.21.15.91:0/1876907227 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cd80caa40 0x7f2cd80caed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.844+0000 7f2cdcfc9700 1 -- 172.21.15.91:0/1876907227 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cd80c7160 msgr2=0x7f2cd80c83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:55.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.844+0000 7f2cdcfc9700 1 --2- 172.21.15.91:0/1876907227 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cd80c7160 0x7f2cd80c83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:55.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.844+0000 7f2cdcfc9700 1 -- 172.21.15.91:0/1876907227 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd80c7920 con 0x7f2cc8003a00 2023-11-24T06:08:55.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.844+0000 7f2cd7fff700 1 --2- 172.21.15.91:0/1876907227 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cd80c7160 0x7f2cd80c83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:55.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.844+0000 7f2cdcfc9700 1 --2- 172.21.15.91:0/1876907227 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2cc8003a00 0x7f2cd815a1f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800c080 tx=0x7f2ccc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:55.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.844+0000 7f2cd57fa700 1 -- 172.21.15.91:0/1876907227 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc00e6c0 con 0x7f2cc8003a00 2023-11-24T06:08:55.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.844+0000 7f2cd57fa700 1 -- 172.21.15.91:0/1876907227 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ccc003ad0 con 0x7f2cc8003a00 2023-11-24T06:08:55.852 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.844+0000 7f2cdf22d700 1 -- 172.21.15.91:0/1876907227 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd80c7c00 con 0x7f2cc8003a00 2023-11-24T06:08:55.853 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.844+0000 7f2cd57fa700 1 -- 172.21.15.91:0/1876907227 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc003e80 con 0x7f2cc8003a00 2023-11-24T06:08:55.853 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.844+0000 7f2cdf22d700 1 -- 172.21.15.91:0/1876907227 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd80c88f0 con 0x7f2cc8003a00 2023-11-24T06:08:55.853 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.845+0000 7f2cdf22d700 1 -- 172.21.15.91:0/1876907227 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd806bc40 con 0x7f2cc8003a00 2023-11-24T06:08:55.853 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.845+0000 7f2cd57fa700 1 -- 172.21.15.91:0/1876907227 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2ccc013470 con 0x7f2cc8003a00 2023-11-24T06:08:55.853 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.846+0000 7f2cd57fa700 1 --2- 172.21.15.91:0/1876907227 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2cbc06aed0 0x7f2cbc06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:55.853 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.846+0000 7f2cd57fa700 1 -- 172.21.15.91:0/1876907227 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2ccc08aa10 con 0x7f2cc8003a00 2023-11-24T06:08:55.853 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.846+0000 7f2cd7fff700 1 --2- 172.21.15.91:0/1876907227 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2cbc06aed0 0x7f2cbc06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:55.853 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.846+0000 7f2cd7fff700 1 --2- 172.21.15.91:0/1876907227 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2cbc06aed0 0x7f2cbc06d390 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4000c00 tx=0x7f2cc4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:55.854 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:55.850+0000 7f2cd57fa700 1 -- 172.21.15.91:0/1876907227 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ccc055ea0 con 0x7f2cc8003a00 2023-11-24T06:08:56.044 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:56.040+0000 7f2cdf22d700 1 -- 172.21.15.91:0/1876907227 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2cd80b4e00 con 0x7f2cc8003a00 2023-11-24T06:08:57.036 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.032+0000 7f2cd57fa700 1 -- 172.21.15.91:0/1876907227 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2ccc017020 con 0x7f2cc8003a00 2023-11-24T06:08:57.040 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.036+0000 7f2cdf22d700 1 -- 172.21.15.91:0/1876907227 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2cbc06aed0 msgr2=0x7f2cbc06d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:57.040 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.036+0000 7f2cdf22d700 1 --2- 172.21.15.91:0/1876907227 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2cbc06aed0 0x7f2cbc06d390 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4000c00 tx=0x7f2cc4009040 comp rx=0 tx=0).stop 2023-11-24T06:08:57.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.036+0000 7f2cdf22d700 1 -- 172.21.15.91:0/1876907227 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2cc8003a00 msgr2=0x7f2cd815a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:57.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.036+0000 7f2cdf22d700 1 --2- 172.21.15.91:0/1876907227 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2cc8003a00 0x7f2cd815a1f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800c080 tx=0x7f2ccc0047c0 comp rx=0 tx=0).stop 2023-11-24T06:08:57.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.036+0000 7f2cdf22d700 1 -- 172.21.15.91:0/1876907227 shutdown_connections 2023-11-24T06:08:57.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.036+0000 7f2cdf22d700 1 --2- 172.21.15.91:0/1876907227 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2cbc06aed0 0x7f2cbc06d390 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:57.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.036+0000 7f2cdf22d700 1 --2- 172.21.15.91:0/1876907227 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2cd80caa40 0x7f2cd80caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:57.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.036+0000 7f2cdf22d700 1 --2- 172.21.15.91:0/1876907227 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2cc8003a00 0x7f2cd815a1f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:57.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.036+0000 7f2cdf22d700 1 --2- 172.21.15.91:0/1876907227 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2cd80c7160 0x7f2cd80c83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:57.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.036+0000 7f2cdf22d700 1 -- 172.21.15.91:0/1876907227 >> 172.21.15.91:0/1876907227 conn(0x7f2cd80b3700 msgr2=0x7f2cd80b2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:57.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.037+0000 7f2cdf22d700 1 -- 172.21.15.91:0/1876907227 shutdown_connections 2023-11-24T06:08:57.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.037+0000 7f2cdf22d700 1 -- 172.21.15.91:0/1876907227 wait complete. 2023-11-24T06:08:57.046 INFO:teuthology.orchestra.run.smithi091.stderr:pool 'cephfs_metadata' created 2023-11-24T06:08:57.058 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-11-24T06:08:57.190 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.186+0000 7fc7510d3700 1 Processor -- start 2023-11-24T06:08:57.190 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.186+0000 7fc7510d3700 1 -- start start 2023-11-24T06:08:57.193 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.187+0000 7fc7510d3700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc74c15bde0 0x7fc74c15c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:57.193 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.187+0000 7fc7510d3700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fc74c0ce5c0 0x7fc74c0cea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:57.193 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.187+0000 7fc7510d3700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc74c0d0a90 con 0x7fc74c0cf2b0 2023-11-24T06:08:57.193 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.187+0000 7fc7510d3700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc74c0cef70 con 0x7fc74c0ce5c0 2023-11-24T06:08:57.193 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.187+0000 7fc7510d3700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc74c06ee10 con 0x7fc74c15bde0 2023-11-24T06:08:57.193 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.187+0000 7fc74a59c700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fc74c0cf2b0 0x7fc74c0d01b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54514/0 (socket says 172.21.15.91:54514) 2023-11-24T06:08:57.193 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.187+0000 7fc74b59e700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fc74c0ce5c0 0x7fc74c0cea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:57.193 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.187+0000 7fc74a59c700 1 -- 172.21.15.91:0/236906676 learned_addr learned my addr 172.21.15.91:0/236906676 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:57.193 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.187+0000 7fc74ad9d700 1 --2- 172.21.15.91:0/236906676 >> v2:172.21.15.94:3300/0 conn(0x7fc74c15bde0 0x7fc74c15c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:57.193 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.188+0000 7fc74b59e700 1 -- 172.21.15.91:0/236906676 >> v2:172.21.15.94:3300/0 conn(0x7fc74c15bde0 msgr2=0x7fc74c15c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:57.194 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.188+0000 7fc74b59e700 1 --2- 172.21.15.91:0/236906676 >> v2:172.21.15.94:3300/0 conn(0x7fc74c15bde0 0x7fc74c15c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:57.194 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.188+0000 7fc74b59e700 1 -- 172.21.15.91:0/236906676 >> v1:172.21.15.91:6789/0 conn(0x7fc74c0cf2b0 legacy=0x7fc74c0d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:57.194 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.188+0000 7fc74b59e700 1 -- 172.21.15.91:0/236906676 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74c06f0f0 con 0x7fc74c0ce5c0 2023-11-24T06:08:57.194 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.188+0000 7fc749d9b700 1 -- 172.21.15.91:0/236906676 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348683730 0 0) 0x7fc74c0d0a90 con 0x7fc74c0cf2b0 2023-11-24T06:08:57.194 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.188+0000 7fc74b59e700 1 --2- 172.21.15.91:0/236906676 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc74c0ce5c0 0x7fc74c0cea30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc74c072370 tx=0x7fc74400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5248dc7211df69f4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:57.194 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.188+0000 7fc749d9b700 1 -- 172.21.15.91:0/236906676 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74400e070 con 0x7fc74c0ce5c0 2023-11-24T06:08:57.194 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.188+0000 7fc749d9b700 1 -- 172.21.15.91:0/236906676 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74400bbd0 con 0x7fc74c0ce5c0 2023-11-24T06:08:57.194 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.188+0000 7fc749d9b700 1 -- 172.21.15.91:0/236906676 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc744012430 con 0x7fc74c0ce5c0 2023-11-24T06:08:57.194 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.188+0000 7fc7510d3700 1 -- 172.21.15.91:0/236906676 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc74c0ce5c0 msgr2=0x7fc74c0cea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:57.194 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.188+0000 7fc7510d3700 1 --2- 172.21.15.91:0/236906676 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc74c0ce5c0 0x7fc74c0cea30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc74c072370 tx=0x7fc74400b330 comp rx=0 tx=0).stop 2023-11-24T06:08:57.194 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.188+0000 7fc7510d3700 1 -- 172.21.15.91:0/236906676 shutdown_connections 2023-11-24T06:08:57.194 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.188+0000 7fc7510d3700 1 --2- 172.21.15.91:0/236906676 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc74c0ce5c0 0x7fc74c0cea30 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:57.195 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.188+0000 7fc7510d3700 1 --2- 172.21.15.91:0/236906676 >> v2:172.21.15.94:3300/0 conn(0x7fc74c15bde0 0x7fc74c15c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:57.195 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.188+0000 7fc7510d3700 1 -- 172.21.15.91:0/236906676 >> 172.21.15.91:0/236906676 conn(0x7fc74c0b3720 msgr2=0x7fc74c0b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:57.195 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.189+0000 7fc7510d3700 1 -- 172.21.15.91:0/236906676 shutdown_connections 2023-11-24T06:08:57.195 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.189+0000 7fc7510d3700 1 -- 172.21.15.91:0/236906676 wait complete. 2023-11-24T06:08:57.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.190+0000 7fc7510d3700 1 Processor -- start 2023-11-24T06:08:57.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.190+0000 7fc7510d3700 1 -- start start 2023-11-24T06:08:57.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc7510d3700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc74c0ce5c0 0x7fc74c15ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:57.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc7510d3700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc74c0cf2b0 0x7fc74c157bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:57.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc7510d3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc74c15bde0 0x7fc74c159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:57.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc7510d3700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c158110 con 0x7fc74c0cf2b0 2023-11-24T06:08:57.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc7510d3700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c15db80 con 0x7fc74c0ce5c0 2023-11-24T06:08:57.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc7510d3700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c15de60 con 0x7fc74c15bde0 2023-11-24T06:08:57.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc74a59c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc74c0cf2b0 0x7fc74c157bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:57.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc74a59c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc74c0cf2b0 0x7fc74c157bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59588/0 (socket says 172.21.15.91:59588) 2023-11-24T06:08:57.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc74a59c700 1 -- 172.21.15.91:0/963546811 learned_addr learned my addr 172.21.15.91:0/963546811 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:57.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc74ad9d700 1 --2- 172.21.15.91:0/963546811 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc74c0ce5c0 0x7fc74c15ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:57.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc74b59e700 1 --2- 172.21.15.91:0/963546811 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc74c15bde0 0x7fc74c159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc74a59c700 1 -- 172.21.15.91:0/963546811 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc74c15bde0 msgr2=0x7fc74c159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc74a59c700 1 --2- 172.21.15.91:0/963546811 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc74c15bde0 0x7fc74c159960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc74a59c700 1 -- 172.21.15.91:0/963546811 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc74c0ce5c0 msgr2=0x7fc74c15ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc74a59c700 1 --2- 172.21.15.91:0/963546811 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc74c0ce5c0 0x7fc74c15ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc74a59c700 1 -- 172.21.15.91:0/963546811 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc744009a70 con 0x7fc74c0cf2b0 2023-11-24T06:08:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc74a59c700 1 --2- 172.21.15.91:0/963546811 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc74c0cf2b0 0x7fc74c157bd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00c540 tx=0x7fc73c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.191+0000 7fc7337fe700 1 -- 172.21.15.91:0/963546811 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c004c80 con 0x7fc74c0cf2b0 2023-11-24T06:08:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.192+0000 7fc7337fe700 1 -- 172.21.15.91:0/963546811 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73c004de0 con 0x7fc74c0cf2b0 2023-11-24T06:08:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.192+0000 7fc7510d3700 1 -- 172.21.15.91:0/963546811 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc74c15e140 con 0x7fc74c0cf2b0 2023-11-24T06:08:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.192+0000 7fc7337fe700 1 -- 172.21.15.91:0/963546811 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c007310 con 0x7fc74c0cf2b0 2023-11-24T06:08:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.192+0000 7fc7510d3700 1 -- 172.21.15.91:0/963546811 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc74c15e700 con 0x7fc74c0cf2b0 2023-11-24T06:08:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.192+0000 7fc7510d3700 1 -- 172.21.15.91:0/963546811 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc74c0c5330 con 0x7fc74c0cf2b0 2023-11-24T06:08:57.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.193+0000 7fc7337fe700 1 -- 172.21.15.91:0/963546811 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc73c007470 con 0x7fc74c0cf2b0 2023-11-24T06:08:57.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.193+0000 7fc7337fe700 1 --2- 172.21.15.91:0/963546811 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc73806aed0 0x7fc73806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:57.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.193+0000 7fc7337fe700 1 -- 172.21.15.91:0/963546811 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fc73c088fc0 con 0x7fc74c0cf2b0 2023-11-24T06:08:57.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.193+0000 7fc74ad9d700 1 --2- 172.21.15.91:0/963546811 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc73806aed0 0x7fc73806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:57.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.194+0000 7fc74ad9d700 1 --2- 172.21.15.91:0/963546811 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc73806aed0 0x7fc73806d390 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc740006b20 tx=0x7fc740008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:57.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.199+0000 7fc7337fe700 1 -- 172.21.15.91:0/963546811 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc73c0bbe90 con 0x7fc74c0cf2b0 2023-11-24T06:08:57.393 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:57.389+0000 7fc7510d3700 1 -- 172.21.15.91:0/963546811 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc74c15e920 con 0x7fc74c0cf2b0 2023-11-24T06:08:58.043 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.040+0000 7fc7337fe700 1 -- 172.21.15.91:0/963546811 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc73c0542e0 con 0x7fc74c0cf2b0 2023-11-24T06:08:58.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.043+0000 7fc7510d3700 1 -- 172.21.15.91:0/963546811 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc73806aed0 msgr2=0x7fc73806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:58.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.043+0000 7fc7510d3700 1 --2- 172.21.15.91:0/963546811 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc73806aed0 0x7fc73806d390 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc740006b20 tx=0x7fc740008040 comp rx=0 tx=0).stop 2023-11-24T06:08:58.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.043+0000 7fc7510d3700 1 -- 172.21.15.91:0/963546811 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc74c0cf2b0 msgr2=0x7fc74c157bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:58.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.043+0000 7fc7510d3700 1 --2- 172.21.15.91:0/963546811 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc74c0cf2b0 0x7fc74c157bd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00c540 tx=0x7fc73c00c850 comp rx=0 tx=0).stop 2023-11-24T06:08:58.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.043+0000 7fc7510d3700 1 -- 172.21.15.91:0/963546811 shutdown_connections 2023-11-24T06:08:58.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.043+0000 7fc7510d3700 1 --2- 172.21.15.91:0/963546811 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc73806aed0 0x7fc73806d390 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:58.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.043+0000 7fc7510d3700 1 --2- 172.21.15.91:0/963546811 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc74c0ce5c0 0x7fc74c15ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:58.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.043+0000 7fc7510d3700 1 --2- 172.21.15.91:0/963546811 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc74c15bde0 0x7fc74c159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:58.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.043+0000 7fc7510d3700 1 --2- 172.21.15.91:0/963546811 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc74c0cf2b0 0x7fc74c157bd0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:58.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.043+0000 7fc7510d3700 1 -- 172.21.15.91:0/963546811 >> 172.21.15.91:0/963546811 conn(0x7fc74c0b3720 msgr2=0x7fc74c0b0700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:58.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.043+0000 7fc7510d3700 1 -- 172.21.15.91:0/963546811 shutdown_connections 2023-11-24T06:08:58.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.043+0000 7fc7510d3700 1 -- 172.21.15.91:0/963546811 wait complete. 2023-11-24T06:08:58.048 INFO:teuthology.orchestra.run.smithi091.stderr:pool 'cephfs_data' created 2023-11-24T06:08:58.069 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-11-24T06:08:58.193 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.190+0000 7f7973312700 1 Processor -- start 2023-11-24T06:08:58.193 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.190+0000 7f7973312700 1 -- start start 2023-11-24T06:08:58.195 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.191+0000 7f7973312700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f796c15bde0 0x7f796c15c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:58.195 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.191+0000 7f7973312700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f796c0ce5c0 0x7f796c0cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:58.196 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.191+0000 7f7973312700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f796c0d0a90 con 0x7f796c0cf2b0 2023-11-24T06:08:58.196 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.191+0000 7f7973312700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796c0cef90 con 0x7f796c0ce5c0 2023-11-24T06:08:58.196 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.191+0000 7f7973312700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796c06ee10 con 0x7f796c15bde0 2023-11-24T06:08:58.196 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.191+0000 7f79718af700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f796c0ce5c0 0x7f796c0cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:58.196 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.191+0000 7f79718af700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f796c0ce5c0 0x7f796c0cea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59598/0 (socket says 172.21.15.91:59598) 2023-11-24T06:08:58.196 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.191+0000 7f79708ad700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f796c0cf2b0 0x7f796c0d01b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54528/0 (socket says 172.21.15.91:54528) 2023-11-24T06:08:58.196 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.191+0000 7f79718af700 1 -- 172.21.15.91:0/125105593 learned_addr learned my addr 172.21.15.91:0/125105593 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:58.196 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.191+0000 7f79710ae700 1 --2- 172.21.15.91:0/125105593 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f796c15bde0 0x7f796c15c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:58.196 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.192+0000 7f79718af700 1 -- 172.21.15.91:0/125105593 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f796c15bde0 msgr2=0x7f796c15c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:58.196 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.192+0000 7f79718af700 1 --2- 172.21.15.91:0/125105593 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f796c15bde0 0x7f796c15c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:58.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.192+0000 7f79718af700 1 -- 172.21.15.91:0/125105593 >> v1:172.21.15.91:6789/0 conn(0x7f796c0cf2b0 legacy=0x7f796c0d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:58.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.192+0000 7f79718af700 1 -- 172.21.15.91:0/125105593 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c06f0f0 con 0x7f796c0ce5c0 2023-11-24T06:08:58.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.192+0000 7f7963fff700 1 -- 172.21.15.91:0/125105593 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623787391 0 0) 0x7f796c0d0a90 con 0x7f796c0cf2b0 2023-11-24T06:08:58.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.192+0000 7f79710ae700 1 --2- 172.21.15.91:0/125105593 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f796c15bde0 0x7f796c15c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:58.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.192+0000 7f79718af700 1 --2- 172.21.15.91:0/125105593 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f796c0ce5c0 0x7f796c0cea50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7964009e50 tx=0x7f796400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92592d71f04386f server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:58.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.192+0000 7f7963fff700 1 -- 172.21.15.91:0/125105593 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796400e070 con 0x7f796c0ce5c0 2023-11-24T06:08:58.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.192+0000 7f7963fff700 1 -- 172.21.15.91:0/125105593 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f796400b9c0 con 0x7f796c0ce5c0 2023-11-24T06:08:58.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.192+0000 7f7963fff700 1 -- 172.21.15.91:0/125105593 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796400bd70 con 0x7f796c0ce5c0 2023-11-24T06:08:58.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.192+0000 7f7973312700 1 -- 172.21.15.91:0/125105593 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f796c0ce5c0 msgr2=0x7f796c0cea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:58.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.192+0000 7f7973312700 1 --2- 172.21.15.91:0/125105593 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f796c0ce5c0 0x7f796c0cea50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7964009e50 tx=0x7f796400b330 comp rx=0 tx=0).stop 2023-11-24T06:08:58.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.192+0000 7f7973312700 1 -- 172.21.15.91:0/125105593 shutdown_connections 2023-11-24T06:08:58.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.192+0000 7f7973312700 1 --2- 172.21.15.91:0/125105593 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f796c0ce5c0 0x7f796c0cea50 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:58.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.192+0000 7f7973312700 1 --2- 172.21.15.91:0/125105593 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f796c15bde0 0x7f796c15c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:58.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.192+0000 7f7973312700 1 -- 172.21.15.91:0/125105593 >> 172.21.15.91:0/125105593 conn(0x7f796c0b3720 msgr2=0x7f796c0b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:58.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.193+0000 7f7973312700 1 -- 172.21.15.91:0/125105593 shutdown_connections 2023-11-24T06:08:58.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.193+0000 7f7973312700 1 -- 172.21.15.91:0/125105593 wait complete. 2023-11-24T06:08:58.201 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.194+0000 7f7973312700 1 Processor -- start 2023-11-24T06:08:58.201 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.194+0000 7f7973312700 1 -- start start 2023-11-24T06:08:58.201 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.194+0000 7f7973312700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f796c0ce5c0 0x7f796c0cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:58.201 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.194+0000 7f7973312700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f796c0cf2b0 0x7f796c0cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:58.201 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.194+0000 7f7973312700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f796c15bde0 0x7f796c1f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:58.201 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.194+0000 7f7973312700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c0cbd00 con 0x7f796c0cf2b0 2023-11-24T06:08:58.201 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.194+0000 7f7973312700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c15ea90 con 0x7f796c0ce5c0 2023-11-24T06:08:58.201 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.194+0000 7f7973312700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796c0c83a0 con 0x7f796c15bde0 2023-11-24T06:08:58.201 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.194+0000 7f79708ad700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f796c0cf2b0 0x7f796c0cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:58.201 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.195+0000 7f79708ad700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f796c0cf2b0 0x7f796c0cb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59602/0 (socket says 172.21.15.91:59602) 2023-11-24T06:08:58.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.195+0000 7f79708ad700 1 -- 172.21.15.91:0/1362909049 learned_addr learned my addr 172.21.15.91:0/1362909049 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:58.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.195+0000 7f79710ae700 1 --2- 172.21.15.91:0/1362909049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f796c0ce5c0 0x7f796c0cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:58.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.195+0000 7f79718af700 1 --2- 172.21.15.91:0/1362909049 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f796c15bde0 0x7f796c1f4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:58.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.195+0000 7f79708ad700 1 -- 172.21.15.91:0/1362909049 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f796c15bde0 msgr2=0x7f796c1f4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:58.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.195+0000 7f79708ad700 1 --2- 172.21.15.91:0/1362909049 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f796c15bde0 0x7f796c1f4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:58.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.195+0000 7f79708ad700 1 -- 172.21.15.91:0/1362909049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f796c0ce5c0 msgr2=0x7f796c0cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:58.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.195+0000 7f79708ad700 1 --2- 172.21.15.91:0/1362909049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f796c0ce5c0 0x7f796c0cacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:58.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.195+0000 7f79708ad700 1 -- 172.21.15.91:0/1362909049 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7964009b30 con 0x7f796c0cf2b0 2023-11-24T06:08:58.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.195+0000 7f79708ad700 1 --2- 172.21.15.91:0/1362909049 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f796c0cf2b0 0x7f796c0cb1f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f796800c540 tx=0x7f796800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:58.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.195+0000 7f7961ffb700 1 -- 172.21.15.91:0/1362909049 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7968004c80 con 0x7f796c0cf2b0 2023-11-24T06:08:58.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.195+0000 7f7961ffb700 1 -- 172.21.15.91:0/1362909049 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7968004de0 con 0x7f796c0cf2b0 2023-11-24T06:08:58.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.195+0000 7f7973312700 1 -- 172.21.15.91:0/1362909049 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796c0c7e50 con 0x7f796c0cf2b0 2023-11-24T06:08:58.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.195+0000 7f7961ffb700 1 -- 172.21.15.91:0/1362909049 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7968007310 con 0x7f796c0cf2b0 2023-11-24T06:08:58.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.195+0000 7f7973312700 1 -- 172.21.15.91:0/1362909049 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796c15e720 con 0x7f796c0cf2b0 2023-11-24T06:08:58.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.196+0000 7f7973312700 1 -- 172.21.15.91:0/1362909049 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796c075480 con 0x7f796c0cf2b0 2023-11-24T06:08:58.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.196+0000 7f7961ffb700 1 -- 172.21.15.91:0/1362909049 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7968007470 con 0x7f796c0cf2b0 2023-11-24T06:08:58.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.197+0000 7f7961ffb700 1 --2- 172.21.15.91:0/1362909049 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f795006aed0 0x7f795006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:58.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.197+0000 7f79710ae700 1 --2- 172.21.15.91:0/1362909049 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f795006aed0 0x7f795006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:58.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.197+0000 7f7961ffb700 1 -- 172.21.15.91:0/1362909049 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f79680892e0 con 0x7f796c0cf2b0 2023-11-24T06:08:58.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.197+0000 7f79710ae700 1 --2- 172.21.15.91:0/1362909049 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f795006aed0 0x7f795006d390 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f796c1f46f0 tx=0x7f795c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:58.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.202+0000 7f7961ffb700 1 -- 172.21.15.91:0/1362909049 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7968054480 con 0x7f796c0cf2b0 2023-11-24T06:08:58.439 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:58.436+0000 7f7973312700 1 -- 172.21.15.91:0/1362909049 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f796c15ebd0 con 0x7f796c0cf2b0 2023-11-24T06:08:59.044 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.042+0000 7f7961ffb700 1 -- 172.21.15.91:0/1362909049 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7968057aa0 con 0x7f796c0cf2b0 2023-11-24T06:08:59.049 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.046+0000 7f7973312700 1 -- 172.21.15.91:0/1362909049 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f795006aed0 msgr2=0x7f795006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.049 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.046+0000 7f7973312700 1 --2- 172.21.15.91:0/1362909049 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f795006aed0 0x7f795006d390 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f796c1f46f0 tx=0x7f795c008040 comp rx=0 tx=0).stop 2023-11-24T06:08:59.049 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.046+0000 7f7973312700 1 -- 172.21.15.91:0/1362909049 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f796c0cf2b0 msgr2=0x7f796c0cb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.049 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.046+0000 7f7973312700 1 --2- 172.21.15.91:0/1362909049 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f796c0cf2b0 0x7f796c0cb1f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f796800c540 tx=0x7f796800c900 comp rx=0 tx=0).stop 2023-11-24T06:08:59.049 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.047+0000 7f7973312700 1 -- 172.21.15.91:0/1362909049 shutdown_connections 2023-11-24T06:08:59.050 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.047+0000 7f7973312700 1 --2- 172.21.15.91:0/1362909049 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f795006aed0 0x7f795006d390 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.050 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.047+0000 7f7973312700 1 --2- 172.21.15.91:0/1362909049 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f796c0ce5c0 0x7f796c0cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.050 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.047+0000 7f7973312700 1 --2- 172.21.15.91:0/1362909049 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f796c15bde0 0x7f796c1f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.050 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.047+0000 7f7973312700 1 --2- 172.21.15.91:0/1362909049 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f796c0cf2b0 0x7f796c0cb1f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.050 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.047+0000 7f7973312700 1 -- 172.21.15.91:0/1362909049 >> 172.21.15.91:0/1362909049 conn(0x7f796c0b3720 msgr2=0x7f796c0b0160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:59.050 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.047+0000 7f7973312700 1 -- 172.21.15.91:0/1362909049 shutdown_connections 2023-11-24T06:08:59.053 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.049+0000 7f7973312700 1 -- 172.21.15.91:0/1362909049 wait complete. 2023-11-24T06:08:59.055 INFO:teuthology.orchestra.run.smithi091.stderr:new fs with metadata pool 4 and data pool 5 2023-11-24T06:08:59.078 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:59.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.197+0000 7f8beb3fe700 1 Processor -- start 2023-11-24T06:08:59.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.197+0000 7f8beb3fe700 1 -- start start 2023-11-24T06:08:59.201 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.198+0000 7f8beb3fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f8be406b620 0x7f8be40b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.198+0000 7f8beb3fe700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8be41629c0 con 0x7f8be40b5ab0 2023-11-24T06:08:59.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.198+0000 7f8beb3fe700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8be4071f50 con 0x7f8be406ee60 2023-11-24T06:08:59.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.198+0000 7f8beb3fe700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8be4072230 con 0x7f8be406b620 2023-11-24T06:08:59.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.198+0000 7f8be919a700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f8be406b620 0x7f8be40b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.198+0000 7f8be8999700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f8be40b5ab0 0x7f8be4161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54556/0 (socket says 172.21.15.91:54556) 2023-11-24T06:08:59.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.198+0000 7f8be8999700 1 -- 172.21.15.91:0/1916566329 learned_addr learned my addr 172.21.15.91:0/1916566329 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:08:59.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.198+0000 7f8be919a700 1 -- 172.21.15.91:0/1916566329 >> v1:172.21.15.94:6789/0 conn(0x7f8be406ee60 legacy=0x7f8be41622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:59.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.198+0000 7f8be919a700 1 -- 172.21.15.91:0/1916566329 >> v1:172.21.15.91:6789/0 conn(0x7f8be40b5ab0 legacy=0x7f8be4161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.198+0000 7f8bdbfff700 1 -- 172.21.15.91:0/1916566329 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093791410 0 0) 0x7f8be41629c0 con 0x7f8be40b5ab0 2023-11-24T06:08:59.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.198+0000 7f8be919a700 1 -- 172.21.15.91:0/1916566329 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be4072510 con 0x7f8be406b620 2023-11-24T06:08:59.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.199+0000 7f8be919a700 1 --2- 172.21.15.91:0/1916566329 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8be406b620 0x7f8be40b3a40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4009840 tx=0x7f8bd4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b8e0eca28cd174 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:59.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.199+0000 7f8bdbfff700 1 -- 172.21.15.91:0/1916566329 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4004230 con 0x7f8be406b620 2023-11-24T06:08:59.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.199+0000 7f8bdbfff700 1 -- 172.21.15.91:0/1916566329 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd4004390 con 0x7f8be406b620 2023-11-24T06:08:59.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.199+0000 7f8bdbfff700 1 -- 172.21.15.91:0/1916566329 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd40036a0 con 0x7f8be406b620 2023-11-24T06:08:59.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.199+0000 7f8beb3fe700 1 -- 172.21.15.91:0/1916566329 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8be406b620 msgr2=0x7f8be40b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.199+0000 7f8beb3fe700 1 --2- 172.21.15.91:0/1916566329 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8be406b620 0x7f8be40b3a40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4009840 tx=0x7f8bd4009b50 comp rx=0 tx=0).stop 2023-11-24T06:08:59.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.199+0000 7f8beb3fe700 1 -- 172.21.15.91:0/1916566329 shutdown_connections 2023-11-24T06:08:59.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.199+0000 7f8beb3fe700 1 --2- 172.21.15.91:0/1916566329 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8be406b620 0x7f8be40b3a40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.199+0000 7f8beb3fe700 1 -- 172.21.15.91:0/1916566329 >> 172.21.15.91:0/1916566329 conn(0x7f8be40b2740 msgr2=0x7f8be40b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:59.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.199+0000 7f8beb3fe700 1 -- 172.21.15.91:0/1916566329 shutdown_connections 2023-11-24T06:08:59.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.199+0000 7f8beb3fe700 1 -- 172.21.15.91:0/1916566329 wait complete. 2023-11-24T06:08:59.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.201+0000 7f8beb3fe700 1 Processor -- start 2023-11-24T06:08:59.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.201+0000 7f8beb3fe700 1 -- start start 2023-11-24T06:08:59.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.201+0000 7f8beb3fe700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8be406b620 0x7f8be40cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.201+0000 7f8beb3fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8be406ee60 0x7f8be41534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.201+0000 7f8beb3fe700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8be40b5ab0 0x7f8be41539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.201+0000 7f8beb3fe700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be4152400 con 0x7f8be406ee60 2023-11-24T06:08:59.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.201+0000 7f8beb3fe700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be41526e0 con 0x7f8be40b5ab0 2023-11-24T06:08:59.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.201+0000 7f8beb3fe700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be41529c0 con 0x7f8be406b620 2023-11-24T06:08:59.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.201+0000 7f8be8999700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8be406ee60 0x7f8be41534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.201+0000 7f8be8999700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8be406ee60 0x7f8be41534a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59630/0 (socket says 172.21.15.91:59630) 2023-11-24T06:08:59.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.201+0000 7f8be8999700 1 -- 172.21.15.91:0/2669235853 learned_addr learned my addr 172.21.15.91:0/2669235853 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:59.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.202+0000 7f8be8999700 1 -- 172.21.15.91:0/2669235853 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8be406b620 msgr2=0x7f8be40cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.202+0000 7f8be919a700 1 --2- 172.21.15.91:0/2669235853 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8be406b620 0x7f8be40cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.202+0000 7f8be999b700 1 --2- 172.21.15.91:0/2669235853 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8be40b5ab0 0x7f8be41539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.202+0000 7f8be8999700 1 --2- 172.21.15.91:0/2669235853 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8be406b620 0x7f8be40cdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.208 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.202+0000 7f8be8999700 1 -- 172.21.15.91:0/2669235853 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8be40b5ab0 msgr2=0x7f8be41539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.208 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.202+0000 7f8be8999700 1 --2- 172.21.15.91:0/2669235853 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8be40b5ab0 0x7f8be41539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.208 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.202+0000 7f8be8999700 1 -- 172.21.15.91:0/2669235853 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd40094a0 con 0x7f8be406ee60 2023-11-24T06:08:59.208 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.202+0000 7f8be8999700 1 --2- 172.21.15.91:0/2669235853 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8be406ee60 0x7f8be41534a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8be000c520 tx=0x7f8be000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:59.208 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.202+0000 7f8bd9ffb700 1 -- 172.21.15.91:0/2669235853 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be0004c80 con 0x7f8be406ee60 2023-11-24T06:08:59.208 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.202+0000 7f8bd9ffb700 1 -- 172.21.15.91:0/2669235853 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be0004de0 con 0x7f8be406ee60 2023-11-24T06:08:59.208 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.202+0000 7f8bd9ffb700 1 -- 172.21.15.91:0/2669235853 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be00072d0 con 0x7f8be406ee60 2023-11-24T06:08:59.208 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.202+0000 7f8beb3fe700 1 -- 172.21.15.91:0/2669235853 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8be4152ca0 con 0x7f8be406ee60 2023-11-24T06:08:59.208 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.202+0000 7f8beb3fe700 1 -- 172.21.15.91:0/2669235853 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8be4151fb0 con 0x7f8be406ee60 2023-11-24T06:08:59.208 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.203+0000 7f8beb3fe700 1 -- 172.21.15.91:0/2669235853 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be40b6d30 con 0x7f8be406ee60 2023-11-24T06:08:59.209 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.203+0000 7f8bd9ffb700 1 -- 172.21.15.91:0/2669235853 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8be0007430 con 0x7f8be406ee60 2023-11-24T06:08:59.209 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.204+0000 7f8bd9ffb700 1 --2- 172.21.15.91:0/2669235853 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8bc806aed0 0x7f8bc806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.209 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.204+0000 7f8be919a700 1 --2- 172.21.15.91:0/2669235853 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8bc806aed0 0x7f8bc806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.209 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.204+0000 7f8bd9ffb700 1 -- 172.21.15.91:0/2669235853 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8be0089380 con 0x7f8be406ee60 2023-11-24T06:08:59.210 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.204+0000 7f8be919a700 1 --2- 172.21.15.91:0/2669235853 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8bc806aed0 0x7f8bc806d390 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4009b90 tx=0x7f8bd4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:59.211 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.209+0000 7f8bd9ffb700 1 -- 172.21.15.91:0/2669235853 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8be0054430 con 0x7f8be406ee60 2023-11-24T06:08:59.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.397+0000 7f8beb3fe700 1 -- 172.21.15.91:0/2669235853 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8be41531c0 con 0x7f8be406ee60 2023-11-24T06:08:59.401 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.399+0000 7f8bd9ffb700 1 -- 172.21.15.91:0/2669235853 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16498 (secure 0 0 0) 0x7f8be41531c0 con 0x7f8be406ee60 2023-11-24T06:08:59.402 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:59.402 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":33,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:58.437638+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:08:56.059841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:08:57.408988+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:59.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.401+0000 7f8beb3fe700 1 -- 172.21.15.91:0/2669235853 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8bc806aed0 msgr2=0x7f8bc806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.401+0000 7f8beb3fe700 1 --2- 172.21.15.91:0/2669235853 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8bc806aed0 0x7f8bc806d390 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8bd4009b90 tx=0x7f8bd4008030 comp rx=0 tx=0).stop 2023-11-24T06:08:59.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.401+0000 7f8beb3fe700 1 -- 172.21.15.91:0/2669235853 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8be406ee60 msgr2=0x7f8be41534a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.401+0000 7f8beb3fe700 1 --2- 172.21.15.91:0/2669235853 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8be406ee60 0x7f8be41534a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8be000c520 tx=0x7f8be000c8e0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.401+0000 7f8beb3fe700 1 -- 172.21.15.91:0/2669235853 shutdown_connections 2023-11-24T06:08:59.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.401+0000 7f8beb3fe700 1 --2- 172.21.15.91:0/2669235853 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8bc806aed0 0x7f8bc806d390 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.401+0000 7f8beb3fe700 1 --2- 172.21.15.91:0/2669235853 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8be40b5ab0 0x7f8be41539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.401+0000 7f8beb3fe700 1 --2- 172.21.15.91:0/2669235853 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8be406b620 0x7f8be40cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.404 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.401+0000 7f8beb3fe700 1 --2- 172.21.15.91:0/2669235853 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8be406ee60 0x7f8be41534a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.401+0000 7f8beb3fe700 1 -- 172.21.15.91:0/2669235853 >> 172.21.15.91:0/2669235853 conn(0x7f8be40b2740 msgr2=0x7f8be40b3830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:59.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.401+0000 7f8beb3fe700 1 -- 172.21.15.91:0/2669235853 shutdown_connections 2023-11-24T06:08:59.405 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.401+0000 7f8beb3fe700 1 -- 172.21.15.91:0/2669235853 wait complete. 2023-11-24T06:08:59.416 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:08:59.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.534+0000 7f3be701e700 1 Processor -- start 2023-11-24T06:08:59.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.534+0000 7f3be701e700 1 -- start start 2023-11-24T06:08:59.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.534+0000 7f3be701e700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3be006b620 0x7f3be00b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.534+0000 7f3be701e700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f3be00b5ab0 0x7f3be00b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.534+0000 7f3be701e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3be01478c0 0x7f3be0161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.534+0000 7f3be701e700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3be0071f50 con 0x7f3be00b5ab0 2023-11-24T06:08:59.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.534+0000 7f3be701e700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3be0072230 con 0x7f3be006b620 2023-11-24T06:08:59.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.534+0000 7f3be701e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be0072510 con 0x7f3be01478c0 2023-11-24T06:08:59.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.535+0000 7f3bdffff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f3be00b5ab0 0x7f3be00b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.535+0000 7f3bdffff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f3be00b5ab0 0x7f3be00b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59636/0 (socket says 172.21.15.91:59636) 2023-11-24T06:08:59.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.535+0000 7f3bdffff700 1 -- 172.21.15.91:0/2486315601 learned_addr learned my addr 172.21.15.91:0/2486315601 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:59.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.535+0000 7f3be55bb700 1 --2- 172.21.15.91:0/2486315601 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3be01478c0 0x7f3be0161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.535+0000 7f3be4dba700 1 --2- 172.21.15.91:0/2486315601 >> v2:172.21.15.94:3300/0 conn(0x7f3be006b620 0x7f3be00b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.535+0000 7f3bdffff700 1 -- 172.21.15.91:0/2486315601 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3be01478c0 msgr2=0x7f3be0161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:59.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.535+0000 7f3bdffff700 1 --2- 172.21.15.91:0/2486315601 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3be01478c0 0x7f3be0161ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.535+0000 7f3bdffff700 1 -- 172.21.15.91:0/2486315601 >> v2:172.21.15.94:3300/0 conn(0x7f3be006b620 msgr2=0x7f3be00b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:59.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.535+0000 7f3bdffff700 1 --2- 172.21.15.91:0/2486315601 >> v2:172.21.15.94:3300/0 conn(0x7f3be006b620 0x7f3be00b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.535+0000 7f3bdffff700 1 -- 172.21.15.91:0/2486315601 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be00727f0 con 0x7f3be00b5ab0 2023-11-24T06:08:59.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.535+0000 7f3be4dba700 1 --2- 172.21.15.91:0/2486315601 >> v2:172.21.15.94:3300/0 conn(0x7f3be006b620 0x7f3be00b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:08:59.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.535+0000 7f3bdffff700 1 --2- 172.21.15.91:0/2486315601 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3be00b5ab0 0x7f3be00b3f80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3be006d2e0 tx=0x7f3bcc009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a019260825bdc7b9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:59.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.535+0000 7f3bdf7fe700 1 -- 172.21.15.91:0/2486315601 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc011830 con 0x7f3be00b5ab0 2023-11-24T06:08:59.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.535+0000 7f3bdf7fe700 1 -- 172.21.15.91:0/2486315601 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bcc011990 con 0x7f3be00b5ab0 2023-11-24T06:08:59.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.535+0000 7f3bdf7fe700 1 -- 172.21.15.91:0/2486315601 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc011d20 con 0x7f3be00b5ab0 2023-11-24T06:08:59.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.536+0000 7f3be701e700 1 -- 172.21.15.91:0/2486315601 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3be00b5ab0 msgr2=0x7f3be00b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.536+0000 7f3be701e700 1 --2- 172.21.15.91:0/2486315601 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3be00b5ab0 0x7f3be00b3f80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3be006d2e0 tx=0x7f3bcc009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.536+0000 7f3be701e700 1 -- 172.21.15.91:0/2486315601 shutdown_connections 2023-11-24T06:08:59.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.536+0000 7f3be701e700 1 --2- 172.21.15.91:0/2486315601 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3be01478c0 0x7f3be0161ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.536+0000 7f3be701e700 1 --2- 172.21.15.91:0/2486315601 >> v2:172.21.15.94:3300/0 conn(0x7f3be006b620 0x7f3be00b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.536+0000 7f3be701e700 1 --2- 172.21.15.91:0/2486315601 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3be00b5ab0 0x7f3be00b3f80 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.536+0000 7f3be701e700 1 -- 172.21.15.91:0/2486315601 >> 172.21.15.91:0/2486315601 conn(0x7f3be00b2740 msgr2=0x7f3be00b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:59.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.536+0000 7f3be701e700 1 -- 172.21.15.91:0/2486315601 shutdown_connections 2023-11-24T06:08:59.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.536+0000 7f3be701e700 1 -- 172.21.15.91:0/2486315601 wait complete. 2023-11-24T06:08:59.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.537+0000 7f3be701e700 1 Processor -- start 2023-11-24T06:08:59.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.537+0000 7f3be701e700 1 -- start start 2023-11-24T06:08:59.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3be701e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3be006b620 0x7f3be00c8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3be701e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3be00b5ab0 0x7f3be01373d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3be701e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3be01478c0 0x7f3be00c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3be701e700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be00c6e60 con 0x7f3be00b5ab0 2023-11-24T06:08:59.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3be701e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be0137910 con 0x7f3be01478c0 2023-11-24T06:08:59.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3be701e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be00c61e0 con 0x7f3be006b620 2023-11-24T06:08:59.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3bdffff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3be00b5ab0 0x7f3be01373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3bdffff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3be00b5ab0 0x7f3be01373d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59642/0 (socket says 172.21.15.91:59642) 2023-11-24T06:08:59.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3bdffff700 1 -- 172.21.15.91:0/2618013621 learned_addr learned my addr 172.21.15.91:0/2618013621 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:59.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3be4dba700 1 --2- 172.21.15.91:0/2618013621 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3be006b620 0x7f3be00c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3be55bb700 1 --2- 172.21.15.91:0/2618013621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3be01478c0 0x7f3be00c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3bdffff700 1 -- 172.21.15.91:0/2618013621 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3be006b620 msgr2=0x7f3be00c8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3bdffff700 1 --2- 172.21.15.91:0/2618013621 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3be006b620 0x7f3be00c8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3bdffff700 1 -- 172.21.15.91:0/2618013621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3be01478c0 msgr2=0x7f3be00c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3bdffff700 1 --2- 172.21.15.91:0/2618013621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3be01478c0 0x7f3be00c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3bdffff700 1 -- 172.21.15.91:0/2618013621 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bcc009550 con 0x7f3be00b5ab0 2023-11-24T06:08:59.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.538+0000 7f3bdffff700 1 --2- 172.21.15.91:0/2618013621 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3be00b5ab0 0x7f3be01373d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc000c00 tx=0x7f3bcc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:59.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.539+0000 7f3bdd7fa700 1 -- 172.21.15.91:0/2618013621 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc004240 con 0x7f3be00b5ab0 2023-11-24T06:08:59.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.539+0000 7f3bdd7fa700 1 -- 172.21.15.91:0/2618013621 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bcc01f070 con 0x7f3be00b5ab0 2023-11-24T06:08:59.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.539+0000 7f3be701e700 1 -- 172.21.15.91:0/2618013621 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be00c64c0 con 0x7f3be00b5ab0 2023-11-24T06:08:59.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.539+0000 7f3bdd7fa700 1 -- 172.21.15.91:0/2618013621 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc0036a0 con 0x7f3be00b5ab0 2023-11-24T06:08:59.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.539+0000 7f3be701e700 1 -- 172.21.15.91:0/2618013621 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be00c6a80 con 0x7f3be00b5ab0 2023-11-24T06:08:59.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.540+0000 7f3be701e700 1 -- 172.21.15.91:0/2618013621 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be0068670 con 0x7f3be00b5ab0 2023-11-24T06:08:59.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.540+0000 7f3bdd7fa700 1 -- 172.21.15.91:0/2618013621 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3bcc021030 con 0x7f3be00b5ab0 2023-11-24T06:08:59.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.540+0000 7f3bdd7fa700 1 --2- 172.21.15.91:0/2618013621 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3bc406aed0 0x7f3bc406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.540+0000 7f3bdd7fa700 1 -- 172.21.15.91:0/2618013621 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3bcc08a710 con 0x7f3be00b5ab0 2023-11-24T06:08:59.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.540+0000 7f3be4dba700 1 --2- 172.21.15.91:0/2618013621 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3bc406aed0 0x7f3bc406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.541+0000 7f3be4dba700 1 --2- 172.21.15.91:0/2618013621 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3bc406aed0 0x7f3bc406d390 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40045f0 tx=0x7f3bd4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:59.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.545+0000 7f3bdd7fa700 1 -- 172.21.15.91:0/2618013621 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3bcc055850 con 0x7f3be00b5ab0 2023-11-24T06:08:59.736 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.733+0000 7f3be701e700 1 -- 172.21.15.91:0/2618013621 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3be006bd40 con 0x7f3be00b5ab0 2023-11-24T06:08:59.737 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.735+0000 7f3bdd7fa700 1 -- 172.21.15.91:0/2618013621 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16498 (secure 0 0 0) 0x7f3bcc015090 con 0x7f3be00b5ab0 2023-11-24T06:08:59.738 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:08:59.738 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":33,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:08:58.437638+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:08:56.059841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:08:57.408988+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:08:59.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.736+0000 7f3be701e700 1 -- 172.21.15.91:0/2618013621 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3bc406aed0 msgr2=0x7f3bc406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.737+0000 7f3be701e700 1 --2- 172.21.15.91:0/2618013621 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3bc406aed0 0x7f3bc406d390 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40045f0 tx=0x7f3bd4008040 comp rx=0 tx=0).stop 2023-11-24T06:08:59.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.737+0000 7f3be701e700 1 -- 172.21.15.91:0/2618013621 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3be00b5ab0 msgr2=0x7f3be01373d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.737+0000 7f3be701e700 1 --2- 172.21.15.91:0/2618013621 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3be00b5ab0 0x7f3be01373d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc000c00 tx=0x7f3bcc004750 comp rx=0 tx=0).stop 2023-11-24T06:08:59.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.737+0000 7f3be701e700 1 -- 172.21.15.91:0/2618013621 shutdown_connections 2023-11-24T06:08:59.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.737+0000 7f3be701e700 1 --2- 172.21.15.91:0/2618013621 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3bc406aed0 0x7f3bc406d390 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.737+0000 7f3be701e700 1 --2- 172.21.15.91:0/2618013621 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3be01478c0 0x7f3be00c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.737+0000 7f3be701e700 1 --2- 172.21.15.91:0/2618013621 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3be006b620 0x7f3be00c8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.737+0000 7f3be701e700 1 --2- 172.21.15.91:0/2618013621 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3be00b5ab0 0x7f3be01373d0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.737+0000 7f3be701e700 1 -- 172.21.15.91:0/2618013621 >> 172.21.15.91:0/2618013621 conn(0x7f3be00b2740 msgr2=0x7f3be00afe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:59.741 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.737+0000 7f3be701e700 1 -- 172.21.15.91:0/2618013621 shutdown_connections 2023-11-24T06:08:59.741 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.737+0000 7f3be701e700 1 -- 172.21.15.91:0/2618013621 wait complete. 2023-11-24T06:08:59.752 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-11-24T06:08:59.876 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.873+0000 7fa2926a8700 1 Processor -- start 2023-11-24T06:08:59.876 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.873+0000 7fa2926a8700 1 -- start start 2023-11-24T06:08:59.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.874+0000 7fa2926a8700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa28c15bde0 0x7fa28c15c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.874+0000 7fa2926a8700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fa28c0cf2b0 0x7fa28c0d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.874+0000 7fa2926a8700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa28c0d0a90 con 0x7fa28c0ce5c0 2023-11-24T06:08:59.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.874+0000 7fa2926a8700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa28c06ee10 con 0x7fa28c0cf2b0 2023-11-24T06:08:59.879 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.874+0000 7fa2926a8700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa28c06f0f0 con 0x7fa28c15bde0 2023-11-24T06:08:59.880 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.874+0000 7fa28b7fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fa28c0cf2b0 0x7fa28c0d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.880 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.874+0000 7fa28b7fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fa28c0cf2b0 0x7fa28c0d01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59648/0 (socket says 172.21.15.91:59648) 2023-11-24T06:08:59.880 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.874+0000 7fa28b7fe700 1 -- 172.21.15.91:0/1729908842 learned_addr learned my addr 172.21.15.91:0/1729908842 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:59.880 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.874+0000 7fa28bfff700 1 --2- 172.21.15.91:0/1729908842 >> v2:172.21.15.94:3300/0 conn(0x7fa28c15bde0 0x7fa28c15c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.880 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.874+0000 7fa28b7fe700 1 -- 172.21.15.91:0/1729908842 >> v2:172.21.15.94:3300/0 conn(0x7fa28c15bde0 msgr2=0x7fa28c15c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:08:59.880 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.874+0000 7fa28b7fe700 1 --2- 172.21.15.91:0/1729908842 >> v2:172.21.15.94:3300/0 conn(0x7fa28c15bde0 0x7fa28c15c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.880 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.874+0000 7fa28b7fe700 1 -- 172.21.15.91:0/1729908842 >> v1:172.21.15.94:6789/0 conn(0x7fa28c0ce5c0 legacy=0x7fa28c0cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.880 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.874+0000 7fa28b7fe700 1 -- 172.21.15.91:0/1729908842 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28c06f3d0 con 0x7fa28c0cf2b0 2023-11-24T06:08:59.880 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.874+0000 7fa28b7fe700 1 --2- 172.21.15.91:0/1729908842 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa28c0cf2b0 0x7fa28c0d01b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa28c072370 tx=0x7fa27c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5246e708110524 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:59.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.875+0000 7fa28affd700 1 -- 172.21.15.91:0/1729908842 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c011830 con 0x7fa28c0cf2b0 2023-11-24T06:08:59.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.875+0000 7fa28affd700 1 -- 172.21.15.91:0/1729908842 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27c011990 con 0x7fa28c0cf2b0 2023-11-24T06:08:59.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.875+0000 7fa28affd700 1 -- 172.21.15.91:0/1729908842 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c011d20 con 0x7fa28c0cf2b0 2023-11-24T06:08:59.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.875+0000 7fa2926a8700 1 -- 172.21.15.91:0/1729908842 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa28c0cf2b0 msgr2=0x7fa28c0d01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.875+0000 7fa2926a8700 1 --2- 172.21.15.91:0/1729908842 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa28c0cf2b0 0x7fa28c0d01b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa28c072370 tx=0x7fa27c009bd0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.875+0000 7fa2926a8700 1 -- 172.21.15.91:0/1729908842 shutdown_connections 2023-11-24T06:08:59.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.875+0000 7fa2926a8700 1 --2- 172.21.15.91:0/1729908842 >> v2:172.21.15.94:3300/0 conn(0x7fa28c15bde0 0x7fa28c15c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.875+0000 7fa2926a8700 1 --2- 172.21.15.91:0/1729908842 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa28c0cf2b0 0x7fa28c0d01b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.875+0000 7fa2926a8700 1 -- 172.21.15.91:0/1729908842 >> 172.21.15.91:0/1729908842 conn(0x7fa28c0b3720 msgr2=0x7fa28c0b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:08:59.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.875+0000 7fa2926a8700 1 -- 172.21.15.91:0/1729908842 shutdown_connections 2023-11-24T06:08:59.881 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.875+0000 7fa2926a8700 1 -- 172.21.15.91:0/1729908842 wait complete. 2023-11-24T06:08:59.882 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.876+0000 7fa2926a8700 1 Processor -- start 2023-11-24T06:08:59.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.877+0000 7fa2926a8700 1 -- start start 2023-11-24T06:08:59.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.877+0000 7fa2926a8700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa28c0ce5c0 0x7fa28c1f3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.877+0000 7fa2926a8700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28c0cf2b0 0x7fa28c15ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.884 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.877+0000 7fa2926a8700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa28c15bde0 0x7fa28c15c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.877+0000 7fa2926a8700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c1f44a0 con 0x7fa28c15bde0 2023-11-24T06:08:59.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.877+0000 7fa2926a8700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c06cf90 con 0x7fa28c0cf2b0 2023-11-24T06:08:59.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.877+0000 7fa2926a8700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c15d3c0 con 0x7fa28c0ce5c0 2023-11-24T06:08:59.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.877+0000 7fa290c45700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa28c15bde0 0x7fa28c15c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.877+0000 7fa290c45700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa28c15bde0 0x7fa28c15c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59658/0 (socket says 172.21.15.91:59658) 2023-11-24T06:08:59.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.877+0000 7fa290c45700 1 -- 172.21.15.91:0/893860403 learned_addr learned my addr 172.21.15.91:0/893860403 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:08:59.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.878+0000 7fa28bfff700 1 --2- 172.21.15.91:0/893860403 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa28c0ce5c0 0x7fa28c1f3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.878+0000 7fa28b7fe700 1 --2- 172.21.15.91:0/893860403 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28c0cf2b0 0x7fa28c15ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.878+0000 7fa290c45700 1 -- 172.21.15.91:0/893860403 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa28c0ce5c0 msgr2=0x7fa28c1f3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.878+0000 7fa290c45700 1 --2- 172.21.15.91:0/893860403 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa28c0ce5c0 0x7fa28c1f3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.878+0000 7fa290c45700 1 -- 172.21.15.91:0/893860403 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28c0cf2b0 msgr2=0x7fa28c15ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:08:59.885 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.878+0000 7fa290c45700 1 --2- 172.21.15.91:0/893860403 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28c0cf2b0 0x7fa28c15ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:08:59.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.878+0000 7fa290c45700 1 -- 172.21.15.91:0/893860403 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27c009550 con 0x7fa28c15bde0 2023-11-24T06:08:59.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.878+0000 7fa28b7fe700 1 --2- 172.21.15.91:0/893860403 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28c0cf2b0 0x7fa28c15ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:08:59.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.878+0000 7fa290c45700 1 --2- 172.21.15.91:0/893860403 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa28c15bde0 0x7fa28c15c940 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa28400cc50 tx=0x7fa28400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:59.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.878+0000 7fa288ff9700 1 -- 172.21.15.91:0/893860403 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28400ad80 con 0x7fa28c15bde0 2023-11-24T06:08:59.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.878+0000 7fa288ff9700 1 -- 172.21.15.91:0/893860403 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28400ce50 con 0x7fa28c15bde0 2023-11-24T06:08:59.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.878+0000 7fa2926a8700 1 -- 172.21.15.91:0/893860403 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28c06c660 con 0x7fa28c15bde0 2023-11-24T06:08:59.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.878+0000 7fa288ff9700 1 -- 172.21.15.91:0/893860403 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa284003c40 con 0x7fa28c15bde0 2023-11-24T06:08:59.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.878+0000 7fa2926a8700 1 -- 172.21.15.91:0/893860403 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28c06cc20 con 0x7fa28c15bde0 2023-11-24T06:08:59.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.879+0000 7fa2926a8700 1 -- 172.21.15.91:0/893860403 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa28c06d100 con 0x7fa28c15bde0 2023-11-24T06:08:59.886 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.879+0000 7fa288ff9700 1 -- 172.21.15.91:0/893860403 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa2840050d0 con 0x7fa28c15bde0 2023-11-24T06:08:59.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.880+0000 7fa288ff9700 1 --2- 172.21.15.91:0/893860403 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa27806aed0 0x7fa27806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:08:59.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.880+0000 7fa288ff9700 1 -- 172.21.15.91:0/893860403 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2840896f0 con 0x7fa28c15bde0 2023-11-24T06:08:59.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.880+0000 7fa28bfff700 1 --2- 172.21.15.91:0/893860403 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa27806aed0 0x7fa27806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:08:59.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.880+0000 7fa28bfff700 1 --2- 172.21.15.91:0/893860403 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa27806aed0 0x7fa27806d390 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa280009ad0 tx=0x7fa280008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:08:59.887 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:08:59.884+0000 7fa288ff9700 1 -- 172.21.15.91:0/893860403 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa2840547a0 con 0x7fa28c15bde0 2023-11-24T06:09:00.128 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:00.125+0000 7fa2926a8700 1 -- 172.21.15.91:0/893860403 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa28c075480 con 0x7fa28c15bde0 2023-11-24T06:09:00.981 INFO:tasks.ceph.mds.a.smithi091.stderr:2023-11-24T06:09:00.976+0000 7f4039daf700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T06:09:01.083 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.080+0000 7fa288ff9700 1 -- 172.21.15.91:0/893860403 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa284057dc0 con 0x7fa28c15bde0 2023-11-24T06:09:01.086 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.083+0000 7fa2926a8700 1 -- 172.21.15.91:0/893860403 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa27806aed0 msgr2=0x7fa27806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:01.086 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.083+0000 7fa2926a8700 1 --2- 172.21.15.91:0/893860403 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa27806aed0 0x7fa27806d390 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa280009ad0 tx=0x7fa280008040 comp rx=0 tx=0).stop 2023-11-24T06:09:01.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.083+0000 7fa2926a8700 1 -- 172.21.15.91:0/893860403 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa28c15bde0 msgr2=0x7fa28c15c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:01.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.083+0000 7fa2926a8700 1 --2- 172.21.15.91:0/893860403 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa28c15bde0 0x7fa28c15c940 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa28400cc50 tx=0x7fa28400a340 comp rx=0 tx=0).stop 2023-11-24T06:09:01.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.083+0000 7fa2926a8700 1 -- 172.21.15.91:0/893860403 shutdown_connections 2023-11-24T06:09:01.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.083+0000 7fa2926a8700 1 --2- 172.21.15.91:0/893860403 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa27806aed0 0x7fa27806d390 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:01.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.083+0000 7fa2926a8700 1 --2- 172.21.15.91:0/893860403 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa28c15bde0 0x7fa28c15c940 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:01.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.083+0000 7fa2926a8700 1 --2- 172.21.15.91:0/893860403 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa28c0ce5c0 0x7fa28c1f3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:01.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.083+0000 7fa2926a8700 1 --2- 172.21.15.91:0/893860403 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa28c0cf2b0 0x7fa28c15ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:01.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.083+0000 7fa2926a8700 1 -- 172.21.15.91:0/893860403 >> 172.21.15.91:0/893860403 conn(0x7fa28c0b3720 msgr2=0x7fa28c0cacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:01.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.083+0000 7fa2926a8700 1 -- 172.21.15.91:0/893860403 shutdown_connections 2023-11-24T06:09:01.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.083+0000 7fa2926a8700 1 -- 172.21.15.91:0/893860403 wait complete. 2023-11-24T06:09:01.101 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-11-24T06:09:01.101 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-11-24T06:09:01.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.227+0000 7fd83a216700 1 Processor -- start 2023-11-24T06:09:01.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.228+0000 7fd83a216700 1 -- start start 2023-11-24T06:09:01.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.228+0000 7fd83a216700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd83406fc50 0x7fd83406eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:01.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.228+0000 7fd83a216700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd8340b0e90 0x7fd83406ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:01.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.228+0000 7fd83a216700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8340d1230 con 0x7fd83415e1c0 2023-11-24T06:09:01.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.228+0000 7fd83a216700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8340d0c70 con 0x7fd8340b0e90 2023-11-24T06:09:01.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.228+0000 7fd83a216700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340d0f50 con 0x7fd83406fc50 2023-11-24T06:09:01.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.228+0000 7fd833fff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd8340b0e90 0x7fd83406ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:01.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.228+0000 7fd833fff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd8340b0e90 0x7fd83406ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59662/0 (socket says 172.21.15.91:59662) 2023-11-24T06:09:01.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.228+0000 7fd833fff700 1 -- 172.21.15.91:0/2293587629 learned_addr learned my addr 172.21.15.91:0/2293587629 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:01.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.229+0000 7fd832ffd700 1 --2- 172.21.15.91:0/2293587629 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd83406fc50 0x7fd83406eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:01.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.229+0000 7fd833fff700 1 -- 172.21.15.91:0/2293587629 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd83406fc50 msgr2=0x7fd83406eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:01.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.229+0000 7fd833fff700 1 --2- 172.21.15.91:0/2293587629 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd83406fc50 0x7fd83406eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:01.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.229+0000 7fd833fff700 1 -- 172.21.15.91:0/2293587629 >> v1:172.21.15.94:6789/0 conn(0x7fd83415e1c0 legacy=0x7fd83406f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:01.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.229+0000 7fd833fff700 1 -- 172.21.15.91:0/2293587629 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83406c210 con 0x7fd8340b0e90 2023-11-24T06:09:01.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.229+0000 7fd833fff700 1 --2- 172.21.15.91:0/2293587629 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd8340b0e90 0x7fd83406ca10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd82c0098f0 tx=0x7fd82c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63aa8c0d94f912e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:01.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.229+0000 7fd8327fc700 1 -- 172.21.15.91:0/2293587629 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c004230 con 0x7fd8340b0e90 2023-11-24T06:09:01.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.229+0000 7fd8327fc700 1 -- 172.21.15.91:0/2293587629 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c004390 con 0x7fd8340b0e90 2023-11-24T06:09:01.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.229+0000 7fd8327fc700 1 -- 172.21.15.91:0/2293587629 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c0036a0 con 0x7fd8340b0e90 2023-11-24T06:09:01.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.230+0000 7fd83a216700 1 -- 172.21.15.91:0/2293587629 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd8340b0e90 msgr2=0x7fd83406ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:01.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.230+0000 7fd83a216700 1 --2- 172.21.15.91:0/2293587629 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd8340b0e90 0x7fd83406ca10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd82c0098f0 tx=0x7fd82c009c00 comp rx=0 tx=0).stop 2023-11-24T06:09:01.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.230+0000 7fd83a216700 1 -- 172.21.15.91:0/2293587629 shutdown_connections 2023-11-24T06:09:01.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.230+0000 7fd83a216700 1 --2- 172.21.15.91:0/2293587629 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd8340b0e90 0x7fd83406ca10 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:01.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.230+0000 7fd83a216700 1 --2- 172.21.15.91:0/2293587629 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd83406fc50 0x7fd83406eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:01.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.230+0000 7fd83a216700 1 -- 172.21.15.91:0/2293587629 >> 172.21.15.91:0/2293587629 conn(0x7fd8340b3720 msgr2=0x7fd8340b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:01.235 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.230+0000 7fd83a216700 1 -- 172.21.15.91:0/2293587629 shutdown_connections 2023-11-24T06:09:01.235 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.230+0000 7fd83a216700 1 -- 172.21.15.91:0/2293587629 wait complete. 2023-11-24T06:09:01.237 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.231+0000 7fd83a216700 1 Processor -- start 2023-11-24T06:09:01.237 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.231+0000 7fd83a216700 1 -- start start 2023-11-24T06:09:01.237 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd83a216700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd83406fc50 0x7fd8341571c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:01.237 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd83a216700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd8340b0e90 0x7fd8340ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:01.237 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd83a216700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd83415e1c0 0x7fd8340cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:01.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd83a216700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd834157700 con 0x7fd83415e1c0 2023-11-24T06:09:01.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd83a216700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340cac00 con 0x7fd83406fc50 2023-11-24T06:09:01.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd83a216700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340caed0 con 0x7fd8340b0e90 2023-11-24T06:09:01.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd833fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd83415e1c0 0x7fd8340cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:01.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd833fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd83415e1c0 0x7fd8340cca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59670/0 (socket says 172.21.15.91:59670) 2023-11-24T06:09:01.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd833fff700 1 -- 172.21.15.91:0/522069204 learned_addr learned my addr 172.21.15.91:0/522069204 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:01.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd832ffd700 1 --2- 172.21.15.91:0/522069204 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd8340b0e90 0x7fd8340ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:01.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd8337fe700 1 --2- 172.21.15.91:0/522069204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd83406fc50 0x7fd8341571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:01.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd833fff700 1 -- 172.21.15.91:0/522069204 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd8340b0e90 msgr2=0x7fd8340ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:01.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd833fff700 1 --2- 172.21.15.91:0/522069204 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd8340b0e90 0x7fd8340ca6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:01.238 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd833fff700 1 -- 172.21.15.91:0/522069204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd83406fc50 msgr2=0x7fd8341571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:01.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd833fff700 1 --2- 172.21.15.91:0/522069204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd83406fc50 0x7fd8341571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:01.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd833fff700 1 -- 172.21.15.91:0/522069204 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82c009550 con 0x7fd83415e1c0 2023-11-24T06:09:01.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd832ffd700 1 --2- 172.21.15.91:0/522069204 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd8340b0e90 0x7fd8340ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:01.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd833fff700 1 --2- 172.21.15.91:0/522069204 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd83415e1c0 0x7fd8340cca10 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009130 tx=0x7fd82c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:01.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd81bfff700 1 -- 172.21.15.91:0/522069204 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c003df0 con 0x7fd83415e1c0 2023-11-24T06:09:01.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd81bfff700 1 -- 172.21.15.91:0/522069204 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c01f070 con 0x7fd83415e1c0 2023-11-24T06:09:01.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd83a216700 1 -- 172.21.15.91:0/522069204 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8340cb1b0 con 0x7fd83415e1c0 2023-11-24T06:09:01.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.232+0000 7fd81bfff700 1 -- 172.21.15.91:0/522069204 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c00b880 con 0x7fd83415e1c0 2023-11-24T06:09:01.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.233+0000 7fd83a216700 1 -- 172.21.15.91:0/522069204 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8340cb770 con 0x7fd83415e1c0 2023-11-24T06:09:01.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.233+0000 7fd83a216700 1 -- 172.21.15.91:0/522069204 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd834075480 con 0x7fd83415e1c0 2023-11-24T06:09:01.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.234+0000 7fd81bfff700 1 -- 172.21.15.91:0/522069204 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd82c021030 con 0x7fd83415e1c0 2023-11-24T06:09:01.239 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.234+0000 7fd81bfff700 1 --2- 172.21.15.91:0/522069204 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd82006aed0 0x7fd82006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:01.240 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.234+0000 7fd81bfff700 1 -- 172.21.15.91:0/522069204 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd82c08ac40 con 0x7fd83415e1c0 2023-11-24T06:09:01.240 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.234+0000 7fd8337fe700 1 --2- 172.21.15.91:0/522069204 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd82006aed0 0x7fd82006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:01.240 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.235+0000 7fd8337fe700 1 --2- 172.21.15.91:0/522069204 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd82006aed0 0x7fd82006d390 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd824007bf0 tx=0x7fd824007240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:01.241 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.239+0000 7fd81bfff700 1 -- 172.21.15.91:0/522069204 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd82c055d80 con 0x7fd83415e1c0 2023-11-24T06:09:01.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:01.475+0000 7fd83a216700 1 -- 172.21.15.91:0/522069204 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd834075480 con 0x7fd83415e1c0 2023-11-24T06:09:01.704 DEBUG:teuthology.orchestra.run.smithi091:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:09:01.706 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:09:02.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.085+0000 7fd81bfff700 1 -- 172.21.15.91:0/522069204 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd82c0593a0 con 0x7fd83415e1c0 2023-11-24T06:09:02.090 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.087+0000 7fd83a216700 1 -- 172.21.15.91:0/522069204 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd82006aed0 msgr2=0x7fd82006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.090 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.087+0000 7fd83a216700 1 --2- 172.21.15.91:0/522069204 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd82006aed0 0x7fd82006d390 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd824007bf0 tx=0x7fd824007240 comp rx=0 tx=0).stop 2023-11-24T06:09:02.090 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.087+0000 7fd83a216700 1 -- 172.21.15.91:0/522069204 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd83415e1c0 msgr2=0x7fd8340cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.091 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.087+0000 7fd83a216700 1 --2- 172.21.15.91:0/522069204 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd83415e1c0 0x7fd8340cca10 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd82c009130 tx=0x7fd82c004750 comp rx=0 tx=0).stop 2023-11-24T06:09:02.091 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.087+0000 7fd83a216700 1 -- 172.21.15.91:0/522069204 shutdown_connections 2023-11-24T06:09:02.091 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.087+0000 7fd83a216700 1 --2- 172.21.15.91:0/522069204 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd82006aed0 0x7fd82006d390 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.091 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.087+0000 7fd83a216700 1 --2- 172.21.15.91:0/522069204 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd83406fc50 0x7fd8341571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.091 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.087+0000 7fd83a216700 1 --2- 172.21.15.91:0/522069204 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd83415e1c0 0x7fd8340cca10 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.091 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.087+0000 7fd83a216700 1 --2- 172.21.15.91:0/522069204 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd8340b0e90 0x7fd8340ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.091 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.087+0000 7fd83a216700 1 -- 172.21.15.91:0/522069204 >> 172.21.15.91:0/522069204 conn(0x7fd8340b3720 msgr2=0x7fd8340b3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:02.091 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.088+0000 7fd83a216700 1 -- 172.21.15.91:0/522069204 shutdown_connections 2023-11-24T06:09:02.091 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.088+0000 7fd83a216700 1 -- 172.21.15.91:0/522069204 wait complete. 2023-11-24T06:09:02.103 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-11-24T06:09:02.103 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:09:02.225 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.223+0000 7f60ed8c6700 1 Processor -- start 2023-11-24T06:09:02.226 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.223+0000 7f60ed8c6700 1 -- start start 2023-11-24T06:09:02.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.223+0000 7f60ed8c6700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f60e815e080 0x7f60e80cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.223+0000 7f60ed8c6700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f60e80d0e50 0x7f60e80d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.223+0000 7f60ed8c6700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e80d1280 con 0x7f60e80d2440 2023-11-24T06:09:02.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.223+0000 7f60ed8c6700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e80d14b0 con 0x7f60e80d0e50 2023-11-24T06:09:02.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.223+0000 7f60ed8c6700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e80d2de0 con 0x7f60e815e080 2023-11-24T06:09:02.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.223+0000 7f60e77fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f60e80d0e50 0x7f60e80d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.224+0000 7f60e77fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f60e80d0e50 0x7f60e80d28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59678/0 (socket says 172.21.15.91:59678) 2023-11-24T06:09:02.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.224+0000 7f60e77fe700 1 -- 172.21.15.91:0/1786790971 learned_addr learned my addr 172.21.15.91:0/1786790971 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:02.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.224+0000 7f60e6ffd700 1 --2- 172.21.15.91:0/1786790971 >> v2:172.21.15.94:3300/0 conn(0x7f60e815e080 0x7f60e80cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.224+0000 7f60e77fe700 1 -- 172.21.15.91:0/1786790971 >> v2:172.21.15.94:3300/0 conn(0x7f60e815e080 msgr2=0x7f60e80cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:02.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.224+0000 7f60e77fe700 1 --2- 172.21.15.91:0/1786790971 >> v2:172.21.15.94:3300/0 conn(0x7f60e815e080 0x7f60e80cfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.224+0000 7f60e77fe700 1 -- 172.21.15.91:0/1786790971 >> v1:172.21.15.94:6789/0 conn(0x7f60e80d2440 legacy=0x7f60e80d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:02.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.224+0000 7f60e77fe700 1 -- 172.21.15.91:0/1786790971 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e80ce9a0 con 0x7f60e80d0e50 2023-11-24T06:09:02.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.224+0000 7f60e77fe700 1 --2- 172.21.15.91:0/1786790971 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f60e80d0e50 0x7f60e80d28a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f60e80b6a80 tx=0x7f60e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae0fbf8f5f968db7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:02.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.224+0000 7f60e5ffb700 1 -- 172.21.15.91:0/1786790971 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e000e070 con 0x7f60e80d0e50 2023-11-24T06:09:02.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.224+0000 7f60e5ffb700 1 -- 172.21.15.91:0/1786790971 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e0003a20 con 0x7f60e80d0e50 2023-11-24T06:09:02.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.224+0000 7f60e5ffb700 1 -- 172.21.15.91:0/1786790971 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e0003dd0 con 0x7f60e80d0e50 2023-11-24T06:09:02.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.225+0000 7f60ed8c6700 1 -- 172.21.15.91:0/1786790971 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f60e80d0e50 msgr2=0x7f60e80d28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.225+0000 7f60ed8c6700 1 --2- 172.21.15.91:0/1786790971 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f60e80d0e50 0x7f60e80d28a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f60e80b6a80 tx=0x7f60e000b330 comp rx=0 tx=0).stop 2023-11-24T06:09:02.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.225+0000 7f60ed8c6700 1 -- 172.21.15.91:0/1786790971 shutdown_connections 2023-11-24T06:09:02.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.225+0000 7f60ed8c6700 1 --2- 172.21.15.91:0/1786790971 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f60e80d0e50 0x7f60e80d28a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.225+0000 7f60ed8c6700 1 --2- 172.21.15.91:0/1786790971 >> v2:172.21.15.94:3300/0 conn(0x7f60e815e080 0x7f60e80cfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.225+0000 7f60ed8c6700 1 -- 172.21.15.91:0/1786790971 >> 172.21.15.91:0/1786790971 conn(0x7f60e809d180 msgr2=0x7f60e80d3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:02.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.225+0000 7f60ed8c6700 1 -- 172.21.15.91:0/1786790971 shutdown_connections 2023-11-24T06:09:02.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.225+0000 7f60ed8c6700 1 -- 172.21.15.91:0/1786790971 wait complete. 2023-11-24T06:09:02.232 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.226+0000 7f60ed8c6700 1 Processor -- start 2023-11-24T06:09:02.232 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.226+0000 7f60ed8c6700 1 -- start start 2023-11-24T06:09:02.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.227+0000 7f60ed8c6700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f60e80d0e50 0x7f60e80b7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.227+0000 7f60ed8c6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60e80d2440 0x7f60e8159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.227+0000 7f60ed8c6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e815e080 0x7f60e806c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.227+0000 7f60ed8c6700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e8157790 con 0x7f60e80d0e50 2023-11-24T06:09:02.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.227+0000 7f60ed8c6700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e81f3ec0 con 0x7f60e80d2440 2023-11-24T06:09:02.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.227+0000 7f60ed8c6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e814b020 con 0x7f60e815e080 2023-11-24T06:09:02.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.227+0000 7f60e6ffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f60e80d0e50 0x7f60e80b7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.227+0000 7f60e6ffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f60e80d0e50 0x7f60e80b7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59680/0 (socket says 172.21.15.91:59680) 2023-11-24T06:09:02.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.227+0000 7f60e6ffd700 1 -- 172.21.15.91:0/153460597 learned_addr learned my addr 172.21.15.91:0/153460597 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:02.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.227+0000 7f60e77fe700 1 --2- 172.21.15.91:0/153460597 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e815e080 0x7f60e806c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.227+0000 7f60e67fc700 1 --2- 172.21.15.91:0/153460597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60e80d2440 0x7f60e8159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.228+0000 7f60e6ffd700 1 -- 172.21.15.91:0/153460597 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e815e080 msgr2=0x7f60e806c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.233 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.228+0000 7f60e6ffd700 1 --2- 172.21.15.91:0/153460597 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e815e080 0x7f60e806c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.228+0000 7f60e6ffd700 1 -- 172.21.15.91:0/153460597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60e80d2440 msgr2=0x7f60e8159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.228+0000 7f60e6ffd700 1 --2- 172.21.15.91:0/153460597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60e80d2440 0x7f60e8159310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.228+0000 7f60e6ffd700 1 -- 172.21.15.91:0/153460597 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e0009b30 con 0x7f60e80d0e50 2023-11-24T06:09:02.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.228+0000 7f60e6ffd700 1 --2- 172.21.15.91:0/153460597 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f60e80d0e50 0x7f60e80b7630 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00d800 tx=0x7f60dc00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:02.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.228+0000 7f60cffff700 1 -- 172.21.15.91:0/153460597 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc0125f0 con 0x7f60e80d0e50 2023-11-24T06:09:02.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.228+0000 7f60cffff700 1 -- 172.21.15.91:0/153460597 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60dc012750 con 0x7f60e80d0e50 2023-11-24T06:09:02.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.228+0000 7f60ed8c6700 1 -- 172.21.15.91:0/153460597 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e814b300 con 0x7f60e80d0e50 2023-11-24T06:09:02.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.228+0000 7f60cffff700 1 -- 172.21.15.91:0/153460597 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc012b00 con 0x7f60e80d0e50 2023-11-24T06:09:02.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.228+0000 7f60ed8c6700 1 -- 172.21.15.91:0/153460597 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e80b6e30 con 0x7f60e80d0e50 2023-11-24T06:09:02.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.229+0000 7f60ed8c6700 1 -- 172.21.15.91:0/153460597 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e8067f60 con 0x7f60e80d0e50 2023-11-24T06:09:02.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.229+0000 7f60cffff700 1 -- 172.21.15.91:0/153460597 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f60dc012c60 con 0x7f60e80d0e50 2023-11-24T06:09:02.234 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.230+0000 7f60cffff700 1 --2- 172.21.15.91:0/153460597 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f60d406aed0 0x7f60d406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.235 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.230+0000 7f60cffff700 1 -- 172.21.15.91:0/153460597 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60dc0899d0 con 0x7f60e80d0e50 2023-11-24T06:09:02.235 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.230+0000 7f60e67fc700 1 --2- 172.21.15.91:0/153460597 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f60d406aed0 0x7f60d406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.235 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.230+0000 7f60e67fc700 1 --2- 172.21.15.91:0/153460597 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f60d406aed0 0x7f60d406d390 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f60e80c7300 tx=0x7f60d8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:02.237 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.234+0000 7f60cffff700 1 -- 172.21.15.91:0/153460597 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f60dc054a90 con 0x7f60e80d0e50 2023-11-24T06:09:02.474 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.471+0000 7f60ed8c6700 1 -- 172.21.15.91:0/153460597 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60e8159850 con 0x7f60e80d0e50 2023-11-24T06:09:02.474 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.471+0000 7f60cffff700 1 -- 172.21.15.91:0/153460597 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4907 (secure 0 0 0) 0x7f60dc0580b0 con 0x7f60e80d0e50 2023-11-24T06:09:02.474 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:02.475 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4706,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1685174039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1685174039},{"type":"v1","addr":"172.21.15.94:6834","nonce":1685174039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4709,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1508344631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1508344631},{"type":"v1","addr":"172.21.15.94:6835","nonce":1508344631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5064,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6836/28114058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6834","nonce":28114058},{"type":"v1","addr":"172.21.15.91:6836","nonce":28114058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:08:58.437951+0000","modified":"2023-11-24T06:09:01.476634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.91:6837/1301930708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":1301930708},{"type":"v1","addr":"172.21.15.91:6837","nonce":1301930708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T06:09:02.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.474+0000 7f60ed8c6700 1 -- 172.21.15.91:0/153460597 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f60d406aed0 msgr2=0x7f60d406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.474+0000 7f60ed8c6700 1 --2- 172.21.15.91:0/153460597 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f60d406aed0 0x7f60d406d390 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f60e80c7300 tx=0x7f60d8009040 comp rx=0 tx=0).stop 2023-11-24T06:09:02.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.474+0000 7f60ed8c6700 1 -- 172.21.15.91:0/153460597 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f60e80d0e50 msgr2=0x7f60e80b7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.474+0000 7f60ed8c6700 1 --2- 172.21.15.91:0/153460597 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f60e80d0e50 0x7f60e80b7630 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00d800 tx=0x7f60dc00dbc0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.474+0000 7f60ed8c6700 1 -- 172.21.15.91:0/153460597 shutdown_connections 2023-11-24T06:09:02.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.474+0000 7f60ed8c6700 1 --2- 172.21.15.91:0/153460597 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f60d406aed0 0x7f60d406d390 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.474+0000 7f60ed8c6700 1 --2- 172.21.15.91:0/153460597 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f60e815e080 0x7f60e806c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.474+0000 7f60ed8c6700 1 --2- 172.21.15.91:0/153460597 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f60e80d0e50 0x7f60e80b7630 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.474+0000 7f60ed8c6700 1 --2- 172.21.15.91:0/153460597 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f60e80d2440 0x7f60e8159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.474+0000 7f60ed8c6700 1 -- 172.21.15.91:0/153460597 >> 172.21.15.91:0/153460597 conn(0x7f60e809d180 msgr2=0x7f60e80d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:02.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.474+0000 7f60ed8c6700 1 -- 172.21.15.91:0/153460597 shutdown_connections 2023-11-24T06:09:02.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.474+0000 7f60ed8c6700 1 -- 172.21.15.91:0/153460597 wait complete. 2023-11-24T06:09:02.478 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 16 2023-11-24T06:09:02.489 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:09:02.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.606+0000 7f82b5fad700 1 Processor -- start 2023-11-24T06:09:02.609 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.606+0000 7f82b5fad700 1 -- start start 2023-11-24T06:09:02.612 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.607+0000 7f82b5fad700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f82b00b5ab0 0x7f82b006e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.607+0000 7f82b5fad700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b0071f50 con 0x7f82b006b620 2023-11-24T06:09:02.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.607+0000 7f82b5fad700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b0072230 con 0x7f82b00b4150 2023-11-24T06:09:02.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.607+0000 7f82b5fad700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82b0072510 con 0x7f82b00b5ab0 2023-11-24T06:09:02.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.607+0000 7f82af7fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f82b006b620 0x7f82b00b3a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54622/0 (socket says 172.21.15.91:54622) 2023-11-24T06:09:02.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.607+0000 7f82af7fe700 1 -- 172.21.15.91:0/442672066 learned_addr learned my addr 172.21.15.91:0/442672066 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:02.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82affff700 1 --2- 172.21.15.91:0/442672066 >> v2:172.21.15.94:3300/0 conn(0x7f82b00b5ab0 0x7f82b006e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82ae7fc700 1 -- 172.21.15.91:0/442672066 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979383092 0 0) 0x7f82b0071f50 con 0x7f82b006b620 2023-11-24T06:09:02.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82ae7fc700 1 -- 172.21.15.91:0/442672066 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c0035b0 con 0x7f82b006b620 2023-11-24T06:09:02.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82ae7fc700 1 -- 172.21.15.91:0/442672066 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2599100916 0 0) 0x7f829c0035b0 con 0x7f82b006b620 2023-11-24T06:09:02.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82ae7fc700 1 -- 172.21.15.91:0/442672066 >> v2:172.21.15.94:3300/0 conn(0x7f82b00b5ab0 msgr2=0x7f82b006e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:02.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82ae7fc700 1 --2- 172.21.15.91:0/442672066 >> v2:172.21.15.94:3300/0 conn(0x7f82b00b5ab0 0x7f82b006e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82ae7fc700 1 -- 172.21.15.91:0/442672066 >> v1:172.21.15.94:6789/0 conn(0x7f82b00b4150 legacy=0x7f82b006de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82ae7fc700 1 -- 172.21.15.91:0/442672066 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b00727f0 con 0x7f82b006b620 2023-11-24T06:09:02.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82ae7fc700 1 -- 172.21.15.91:0/442672066 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f8298001670 con 0x7f82b006b620 2023-11-24T06:09:02.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82ae7fc700 1 -- 172.21.15.91:0/442672066 >> v1:172.21.15.91:6789/0 conn(0x7f82b006b620 legacy=0x7f82b00b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82ae7fc700 1 --2- 172.21.15.91:0/442672066 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f829c003a00 0x7f829c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82ae7fc700 1 --2- 172.21.15.91:0/442672066 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f829c004850 0x7f829c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82ae7fc700 1 --2- 172.21.15.91:0/442672066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f829c005210 0x7f829c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82ae7fc700 1 -- 172.21.15.91:0/442672066 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b0072510 con 0x7f829c004850 2023-11-24T06:09:02.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82ae7fc700 1 -- 172.21.15.91:0/442672066 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c00c0b0 con 0x7f829c005210 2023-11-24T06:09:02.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82ae7fc700 1 -- 172.21.15.91:0/442672066 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c00c270 con 0x7f829c003a00 2023-11-24T06:09:02.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82aeffd700 1 --2- 172.21.15.91:0/442672066 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f829c004850 0x7f829c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82af7fe700 1 --2- 172.21.15.91:0/442672066 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f829c003a00 0x7f829c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82affff700 1 --2- 172.21.15.91:0/442672066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f829c005210 0x7f829c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82aeffd700 1 -- 172.21.15.91:0/442672066 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f829c003a00 msgr2=0x7f829c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:02.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82aeffd700 1 --2- 172.21.15.91:0/442672066 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f829c003a00 0x7f829c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82aeffd700 1 -- 172.21.15.91:0/442672066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f829c005210 msgr2=0x7f829c00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:02.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82aeffd700 1 --2- 172.21.15.91:0/442672066 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f829c005210 0x7f829c00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82affff700 1 -- 172.21.15.91:0/442672066 reap_dead start 2023-11-24T06:09:02.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82aeffd700 1 -- 172.21.15.91:0/442672066 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c0035b0 con 0x7f829c004850 2023-11-24T06:09:02.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82af7fe700 1 --2- 172.21.15.91:0/442672066 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f829c003a00 0x7f829c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:02.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.608+0000 7f82aeffd700 1 --2- 172.21.15.91:0/442672066 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f829c004850 0x7f829c004cd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f82a00017c0 tx=0x7f829c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e241db301a36a618 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:02.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.609+0000 7f82ae7fc700 1 -- 172.21.15.91:0/442672066 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a00052d0 con 0x7f829c004850 2023-11-24T06:09:02.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.609+0000 7f82ae7fc700 1 -- 172.21.15.91:0/442672066 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a0005450 con 0x7f829c004850 2023-11-24T06:09:02.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.609+0000 7f82ae7fc700 1 -- 172.21.15.91:0/442672066 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a00038c0 con 0x7f829c004850 2023-11-24T06:09:02.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.609+0000 7f82b5fad700 1 -- 172.21.15.91:0/442672066 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f829c004850 msgr2=0x7f829c004cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.609+0000 7f82b5fad700 1 --2- 172.21.15.91:0/442672066 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f829c004850 0x7f829c004cd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f82a00017c0 tx=0x7f829c005880 comp rx=0 tx=0).stop 2023-11-24T06:09:02.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.609+0000 7f82b5fad700 1 -- 172.21.15.91:0/442672066 shutdown_connections 2023-11-24T06:09:02.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.609+0000 7f82b5fad700 1 --2- 172.21.15.91:0/442672066 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f829c004850 0x7f829c004cd0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.609+0000 7f82b5fad700 1 -- 172.21.15.91:0/442672066 >> 172.21.15.91:0/442672066 conn(0x7f82b00b2740 msgr2=0x7f82b00b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:02.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.609+0000 7f82b5fad700 1 -- 172.21.15.91:0/442672066 shutdown_connections 2023-11-24T06:09:02.616 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.609+0000 7f82b5fad700 1 -- 172.21.15.91:0/442672066 wait complete. 2023-11-24T06:09:02.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.610+0000 7f82b5fad700 1 Processor -- start 2023-11-24T06:09:02.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.611+0000 7f82b5fad700 1 -- start start 2023-11-24T06:09:02.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.611+0000 7f82b5fad700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f829c004850 0x7f82b00caa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.611+0000 7f82b5fad700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82b01e4ca0 0x7f82b01e5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.611+0000 7f82b5fad700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f82b00c6430 0x7f82b00c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.611+0000 7f82b5fad700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b00c7480 con 0x7f82b00c6430 2023-11-24T06:09:02.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.611+0000 7f82b5fad700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b00c71a0 con 0x7f82b01e4ca0 2023-11-24T06:09:02.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.611+0000 7f82b5fad700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b0136380 con 0x7f829c004850 2023-11-24T06:09:02.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.611+0000 7f82affff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f82b00c6430 0x7f82b00c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.611+0000 7f82affff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f82b00c6430 0x7f82b00c68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59704/0 (socket says 172.21.15.91:59704) 2023-11-24T06:09:02.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.611+0000 7f82affff700 1 -- 172.21.15.91:0/1905789324 learned_addr learned my addr 172.21.15.91:0/1905789324 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:02.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.612+0000 7f82aeffd700 1 --2- 172.21.15.91:0/1905789324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82b01e4ca0 0x7f82b01e5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.612+0000 7f82af7fe700 1 --2- 172.21.15.91:0/1905789324 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f829c004850 0x7f82b00caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.612+0000 7f82affff700 1 -- 172.21.15.91:0/1905789324 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f829c004850 msgr2=0x7f82b00caa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.612+0000 7f82affff700 1 --2- 172.21.15.91:0/1905789324 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f829c004850 0x7f82b00caa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.612+0000 7f82affff700 1 -- 172.21.15.91:0/1905789324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82b01e4ca0 msgr2=0x7f82b01e5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.612+0000 7f82affff700 1 --2- 172.21.15.91:0/1905789324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82b01e4ca0 0x7f82b01e5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.612+0000 7f82affff700 1 -- 172.21.15.91:0/1905789324 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b0136660 con 0x7f82b00c6430 2023-11-24T06:09:02.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.612+0000 7f82af7fe700 1 --2- 172.21.15.91:0/1905789324 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f829c004850 0x7f82b00caa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:02.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.612+0000 7f82affff700 1 --2- 172.21.15.91:0/1905789324 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f82b00c6430 0x7f82b00c68b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f82b006bff0 tx=0x7f82a8011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:02.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.612+0000 7f82a7fff700 1 -- 172.21.15.91:0/1905789324 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a800e040 con 0x7f82b00c6430 2023-11-24T06:09:02.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.612+0000 7f82a7fff700 1 -- 172.21.15.91:0/1905789324 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a800f040 con 0x7f82b00c6430 2023-11-24T06:09:02.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.612+0000 7f82b5fad700 1 -- 172.21.15.91:0/1905789324 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b0136940 con 0x7f82b00c6430 2023-11-24T06:09:02.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.612+0000 7f82a7fff700 1 -- 172.21.15.91:0/1905789324 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a800fc40 con 0x7f82b00c6430 2023-11-24T06:09:02.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.612+0000 7f82b5fad700 1 -- 172.21.15.91:0/1905789324 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b00c6df0 con 0x7f82b00c6430 2023-11-24T06:09:02.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.613+0000 7f82b5fad700 1 -- 172.21.15.91:0/1905789324 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b0136cb0 con 0x7f82b00c6430 2023-11-24T06:09:02.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.614+0000 7f82a7fff700 1 -- 172.21.15.91:0/1905789324 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f82a8015430 con 0x7f82b00c6430 2023-11-24T06:09:02.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.614+0000 7f82a7fff700 1 --2- 172.21.15.91:0/1905789324 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f829006aed0 0x7f829006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.614+0000 7f82a7fff700 1 -- 172.21.15.91:0/1905789324 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82a8089e30 con 0x7f82b00c6430 2023-11-24T06:09:02.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.614+0000 7f82af7fe700 1 --2- 172.21.15.91:0/1905789324 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f829006aed0 0x7f829006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.614+0000 7f82af7fe700 1 --2- 172.21.15.91:0/1905789324 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f829006aed0 0x7f829006d390 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8298001c90 tx=0x7f8298009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:02.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.618+0000 7f82a7fff700 1 -- 172.21.15.91:0/1905789324 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82a8054ee0 con 0x7f82b00c6430 2023-11-24T06:09:02.811 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.808+0000 7f82b5fad700 1 -- 172.21.15.91:0/1905789324 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82b0068670 con 0x7f82b00c6430 2023-11-24T06:09:02.812 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.810+0000 7f82a7fff700 1 -- 172.21.15.91:0/1905789324 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16485 (secure 0 0 0) 0x7f82a8058500 con 0x7f82b00c6430 2023-11-24T06:09:02.813 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:02.813 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":34,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:09:00.042725+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:08:56.059841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:08:57.408988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:09:02.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.812+0000 7f82b5fad700 1 -- 172.21.15.91:0/1905789324 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f829006aed0 msgr2=0x7f829006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.812+0000 7f82b5fad700 1 --2- 172.21.15.91:0/1905789324 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f829006aed0 0x7f829006d390 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8298001c90 tx=0x7f8298009040 comp rx=0 tx=0).stop 2023-11-24T06:09:02.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.812+0000 7f82b5fad700 1 -- 172.21.15.91:0/1905789324 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f82b00c6430 msgr2=0x7f82b00c68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.812+0000 7f82b5fad700 1 --2- 172.21.15.91:0/1905789324 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f82b00c6430 0x7f82b00c68b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f82b006bff0 tx=0x7f82a8011e20 comp rx=0 tx=0).stop 2023-11-24T06:09:02.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.812+0000 7f82b5fad700 1 -- 172.21.15.91:0/1905789324 shutdown_connections 2023-11-24T06:09:02.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.812+0000 7f82b5fad700 1 --2- 172.21.15.91:0/1905789324 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f829006aed0 0x7f829006d390 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.812+0000 7f82b5fad700 1 --2- 172.21.15.91:0/1905789324 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f82b00c6430 0x7f82b00c68b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.812+0000 7f82b5fad700 1 --2- 172.21.15.91:0/1905789324 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f829c004850 0x7f82b00caa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.812+0000 7f82b5fad700 1 --2- 172.21.15.91:0/1905789324 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f82b01e4ca0 0x7f82b01e5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.812+0000 7f82b5fad700 1 -- 172.21.15.91:0/1905789324 >> 172.21.15.91:0/1905789324 conn(0x7f82b00b2740 msgr2=0x7f82b00b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:02.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.812+0000 7f82b5fad700 1 -- 172.21.15.91:0/1905789324 shutdown_connections 2023-11-24T06:09:02.816 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.812+0000 7f82b5fad700 1 -- 172.21.15.91:0/1905789324 wait complete. 2023-11-24T06:09:02.827 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-11-24T06:09:02.827 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:09:02.949 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.944+0000 7f0bd8065700 1 Processor -- start 2023-11-24T06:09:02.949 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.944+0000 7f0bd8065700 1 -- start start 2023-11-24T06:09:02.949 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.945+0000 7f0bd8065700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0bd01447e0 0x7f0bd0161b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.949 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.945+0000 7f0bd8065700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f0bd006b6e0 0x7f0bd006e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.949 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.945+0000 7f0bd8065700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd00b3530 con 0x7f0bd0162070 2023-11-24T06:09:02.949 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.945+0000 7f0bd8065700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd006e210 con 0x7f0bd006b6e0 2023-11-24T06:09:02.949 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.945+0000 7f0bd8065700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd006db70 con 0x7f0bd01447e0 2023-11-24T06:09:02.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.945+0000 7f0bd5600700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f0bd0162070 0x7f0bd006afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54642/0 (socket says 172.21.15.91:54642) 2023-11-24T06:09:02.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.945+0000 7f0bd5600700 1 -- 172.21.15.91:0/1685525859 learned_addr learned my addr 172.21.15.91:0/1685525859 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:02.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.945+0000 7f0bd6602700 1 --2- 172.21.15.91:0/1685525859 >> v2:172.21.15.91:3300/0 conn(0x7f0bd006b6e0 0x7f0bd006e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.945+0000 7f0bd5e01700 1 --2- 172.21.15.91:0/1685525859 >> v2:172.21.15.94:3300/0 conn(0x7f0bd01447e0 0x7f0bd0161b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.945+0000 7f0bd4dff700 1 -- 172.21.15.91:0/1685525859 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094815906 0 0) 0x7f0bd00b3530 con 0x7f0bd0162070 2023-11-24T06:09:02.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.946+0000 7f0bd4dff700 1 -- 172.21.15.91:0/1685525859 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bbc0035b0 con 0x7f0bd0162070 2023-11-24T06:09:02.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.946+0000 7f0bd6602700 1 -- 172.21.15.91:0/1685525859 >> v2:172.21.15.94:3300/0 conn(0x7f0bd01447e0 msgr2=0x7f0bd0161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:02.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.946+0000 7f0bd6602700 1 --2- 172.21.15.91:0/1685525859 >> v2:172.21.15.94:3300/0 conn(0x7f0bd01447e0 0x7f0bd0161b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.946+0000 7f0bd6602700 1 -- 172.21.15.91:0/1685525859 >> v1:172.21.15.91:6789/0 conn(0x7f0bd0162070 legacy=0x7f0bd006afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.946+0000 7f0bd6602700 1 -- 172.21.15.91:0/1685525859 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd01626a0 con 0x7f0bd006b6e0 2023-11-24T06:09:02.950 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.946+0000 7f0bd6602700 1 --2- 172.21.15.91:0/1685525859 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0bd006b6e0 0x7f0bd006e630 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0072f50 tx=0x7f0bc800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a23d313e397cbe server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:02.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.946+0000 7f0bd4dff700 1 -- 172.21.15.91:0/1685525859 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc800e070 con 0x7f0bd006b6e0 2023-11-24T06:09:02.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.946+0000 7f0bd4dff700 1 -- 172.21.15.91:0/1685525859 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc800bb20 con 0x7f0bd006b6e0 2023-11-24T06:09:02.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.946+0000 7f0bd4dff700 1 -- 172.21.15.91:0/1685525859 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc800be90 con 0x7f0bd006b6e0 2023-11-24T06:09:02.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.947+0000 7f0bd8065700 1 -- 172.21.15.91:0/1685525859 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0bd006b6e0 msgr2=0x7f0bd006e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.947+0000 7f0bd8065700 1 --2- 172.21.15.91:0/1685525859 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0bd006b6e0 0x7f0bd006e630 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0072f50 tx=0x7f0bc800b330 comp rx=0 tx=0).stop 2023-11-24T06:09:02.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.947+0000 7f0bd8065700 1 -- 172.21.15.91:0/1685525859 shutdown_connections 2023-11-24T06:09:02.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.947+0000 7f0bd8065700 1 --2- 172.21.15.91:0/1685525859 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0bd006b6e0 0x7f0bd006e630 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.947+0000 7f0bd8065700 1 --2- 172.21.15.91:0/1685525859 >> v2:172.21.15.94:3300/0 conn(0x7f0bd01447e0 0x7f0bd0161b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.947+0000 7f0bd8065700 1 -- 172.21.15.91:0/1685525859 >> 172.21.15.91:0/1685525859 conn(0x7f0bd00b2740 msgr2=0x7f0bd00b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:02.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.947+0000 7f0bd8065700 1 -- 172.21.15.91:0/1685525859 shutdown_connections 2023-11-24T06:09:02.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.947+0000 7f0bd8065700 1 -- 172.21.15.91:0/1685525859 wait complete. 2023-11-24T06:09:02.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.948+0000 7f0bd8065700 1 Processor -- start 2023-11-24T06:09:02.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.948+0000 7f0bd8065700 1 -- start start 2023-11-24T06:09:02.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd8065700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0bd006b6e0 0x7f0bd013e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd8065700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bd01447e0 0x7f0bd015ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd8065700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0bd0162070 0x7f0bd00ada10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd8065700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd0144ea0 con 0x7f0bd006b6e0 2023-11-24T06:09:02.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd8065700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd015f7f0 con 0x7f0bd01447e0 2023-11-24T06:09:02.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd8065700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd015fad0 con 0x7f0bd0162070 2023-11-24T06:09:02.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd5e01700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0bd006b6e0 0x7f0bd013e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd5e01700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0bd006b6e0 0x7f0bd013e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59714/0 (socket says 172.21.15.91:59714) 2023-11-24T06:09:02.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd5e01700 1 -- 172.21.15.91:0/2580635240 learned_addr learned my addr 172.21.15.91:0/2580635240 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:02.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd6602700 1 --2- 172.21.15.91:0/2580635240 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0bd0162070 0x7f0bd00ada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd5600700 1 --2- 172.21.15.91:0/2580635240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bd01447e0 0x7f0bd015ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd5e01700 1 -- 172.21.15.91:0/2580635240 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0bd0162070 msgr2=0x7f0bd00ada10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd5e01700 1 --2- 172.21.15.91:0/2580635240 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0bd0162070 0x7f0bd00ada10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd5e01700 1 -- 172.21.15.91:0/2580635240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bd01447e0 msgr2=0x7f0bd015ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:02.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd5e01700 1 --2- 172.21.15.91:0/2580635240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bd01447e0 0x7f0bd015ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:02.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd5e01700 1 -- 172.21.15.91:0/2580635240 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc8009a70 con 0x7f0bd006b6e0 2023-11-24T06:09:02.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd5600700 1 --2- 172.21.15.91:0/2580635240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bd01447e0 0x7f0bd015ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:02.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd5e01700 1 --2- 172.21.15.91:0/2580635240 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0bd006b6e0 0x7f0bd013e2e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc00e9b0 tx=0x7f0bcc00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:02.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bc67fc700 1 -- 172.21.15.91:0/2580635240 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc009710 con 0x7f0bd006b6e0 2023-11-24T06:09:02.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bc67fc700 1 -- 172.21.15.91:0/2580635240 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc00c850 con 0x7f0bd006b6e0 2023-11-24T06:09:02.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bd8065700 1 -- 172.21.15.91:0/2580635240 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd015fdb0 con 0x7f0bd006b6e0 2023-11-24T06:09:02.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.949+0000 7f0bc67fc700 1 -- 172.21.15.91:0/2580635240 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc00cbc0 con 0x7f0bd006b6e0 2023-11-24T06:09:02.957 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.950+0000 7f0bd8065700 1 -- 172.21.15.91:0/2580635240 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd013e820 con 0x7f0bd006b6e0 2023-11-24T06:09:02.957 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.950+0000 7f0bd8065700 1 -- 172.21.15.91:0/2580635240 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd0068670 con 0x7f0bd006b6e0 2023-11-24T06:09:02.957 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.951+0000 7f0bc67fc700 1 -- 172.21.15.91:0/2580635240 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0bcc00cd20 con 0x7f0bd006b6e0 2023-11-24T06:09:02.957 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.951+0000 7f0bc67fc700 1 --2- 172.21.15.91:0/2580635240 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0bb406af00 0x7f0bb406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:02.957 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.951+0000 7f0bc67fc700 1 -- 172.21.15.91:0/2580635240 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0bcc0896d0 con 0x7f0bd006b6e0 2023-11-24T06:09:02.957 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.951+0000 7f0bd5600700 1 --2- 172.21.15.91:0/2580635240 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0bb406af00 0x7f0bb406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:02.957 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.952+0000 7f0bd5600700 1 --2- 172.21.15.91:0/2580635240 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0bb406af00 0x7f0bb406d3c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0147e10 tx=0x7f0bc0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:02.958 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:02.956+0000 7f0bc67fc700 1 -- 172.21.15.91:0/2580635240 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0bcc054780 con 0x7f0bd006b6e0 2023-11-24T06:09:03.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.142+0000 7f0bd8065700 1 -- 172.21.15.91:0/2580635240 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0bd0160270 con 0x7f0bb406af00 2023-11-24T06:09:03.152 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.150+0000 7f0bc67fc700 1 -- 172.21.15.91:0/2580635240 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+276680 (secure 0 0 0) 0x7f0bd0160270 con 0x7f0bb406af00 2023-11-24T06:09:03.156 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:03.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.153+0000 7f0bd8065700 1 -- 172.21.15.91:0/2580635240 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0bb406af00 msgr2=0x7f0bb406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:03.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.153+0000 7f0bd8065700 1 --2- 172.21.15.91:0/2580635240 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0bb406af00 0x7f0bb406d3c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0147e10 tx=0x7f0bc0009040 comp rx=0 tx=0).stop 2023-11-24T06:09:03.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.153+0000 7f0bd8065700 1 -- 172.21.15.91:0/2580635240 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0bd006b6e0 msgr2=0x7f0bd013e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:03.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.153+0000 7f0bd8065700 1 --2- 172.21.15.91:0/2580635240 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0bd006b6e0 0x7f0bd013e2e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc00e9b0 tx=0x7f0bcc00ed70 comp rx=0 tx=0).stop 2023-11-24T06:09:03.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.153+0000 7f0bd8065700 1 -- 172.21.15.91:0/2580635240 shutdown_connections 2023-11-24T06:09:03.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.153+0000 7f0bd8065700 1 --2- 172.21.15.91:0/2580635240 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0bb406af00 0x7f0bb406d3c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.153+0000 7f0bd8065700 1 --2- 172.21.15.91:0/2580635240 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0bd0162070 0x7f0bd00ada10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.153+0000 7f0bd8065700 1 --2- 172.21.15.91:0/2580635240 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0bd006b6e0 0x7f0bd013e2e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.153+0000 7f0bd8065700 1 --2- 172.21.15.91:0/2580635240 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0bd01447e0 0x7f0bd015ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.153+0000 7f0bd8065700 1 -- 172.21.15.91:0/2580635240 >> 172.21.15.91:0/2580635240 conn(0x7f0bd00b2740 msgr2=0x7f0bd00b1740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:03.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.153+0000 7f0bd8065700 1 -- 172.21.15.91:0/2580635240 shutdown_connections 2023-11-24T06:09:03.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.153+0000 7f0bd8065700 1 -- 172.21.15.91:0/2580635240 wait complete. 2023-11-24T06:09:03.160 INFO:teuthology.orchestra.run.smithi091.stderr:dumped all 2023-11-24T06:09:03.168 INFO:teuthology.orchestra.run.smithi091.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2023-11-24T06:09:01.409243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219976,"kb_used_data":3808,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754744,"statfs":{"total":773094113280,"available":772868857856,"internally_reserved":0,"allocated":3899392,"data_stored":2206354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.374220"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.036675+0000","last_change":"2023-11-24T06:08:57.036675+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.036675+0000","last_undegraded":"2023-11-24T06:08:57.036675+0000","last_fullsized":"2023-11-24T06:08:57.036675+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.037677+0000","last_change":"2023-11-24T06:08:57.037677+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.037677+0000","last_undegraded":"2023-11-24T06:08:57.037677+0000","last_fullsized":"2023-11-24T06:08:57.037677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.040679+0000","last_change":"2023-11-24T06:08:57.040679+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.040679+0000","last_undegraded":"2023-11-24T06:08:57.040679+0000","last_fullsized":"2023-11-24T06:08:57.040679+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.038437+0000","last_change":"2023-11-24T06:08:57.038437+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.038437+0000","last_undegraded":"2023-11-24T06:08:57.038437+0000","last_fullsized":"2023-11-24T06:08:57.038437+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.040249+0000","last_change":"2023-11-24T06:08:57.040249+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.040249+0000","last_undegraded":"2023-11-24T06:08:57.040249+0000","last_fullsized":"2023-11-24T06:08:57.040249+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.039108+0000","last_change":"2023-11-24T06:08:57.039108+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.039108+0000","last_undegraded":"2023-11-24T06:08:57.039108+0000","last_fullsized":"2023-11-24T06:08:57.039108+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.042304+0000","last_change":"2023-11-24T06:08:57.042304+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.042304+0000","last_undegraded":"2023-11-24T06:08:57.042304+0000","last_fullsized":"2023-11-24T06:08:57.042304+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.038913+0000","last_change":"2023-11-24T06:08:57.038913+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.038913+0000","last_undegraded":"2023-11-24T06:08:57.038913+0000","last_fullsized":"2023-11-24T06:08:57.038913+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.041826+0000","last_change":"2023-11-24T06:08:57.041826+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.041826+0000","last_undegraded":"2023-11-24T06:08:57.041826+0000","last_fullsized":"2023-11-24T06:08:57.041826+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.037178+0000","last_change":"2023-11-24T06:08:57.037178+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.037178+0000","last_undegraded":"2023-11-24T06:08:57.037178+0000","last_fullsized":"2023-11-24T06:08:57.037178+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.038821+0000","last_change":"2023-11-24T06:08:57.038821+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.038821+0000","last_undegraded":"2023-11-24T06:08:57.038821+0000","last_fullsized":"2023-11-24T06:08:57.038821+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.041383+0000","last_change":"2023-11-24T06:08:57.041383+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.041383+0000","last_undegraded":"2023-11-24T06:08:57.041383+0000","last_fullsized":"2023-11-24T06:08:57.041383+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.039795+0000","last_change":"2023-11-24T06:08:57.039795+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.039795+0000","last_undegraded":"2023-11-24T06:08:57.039795+0000","last_fullsized":"2023-11-24T06:08:57.039795+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.039963+0000","last_change":"2023-11-24T06:08:57.039963+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.039963+0000","last_undegraded":"2023-11-24T06:08:57.039963+0000","last_fullsized":"2023-11-24T06:08:57.039963+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.041810+0000","last_change":"2023-11-24T06:08:57.041810+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.041810+0000","last_undegraded":"2023-11-24T06:08:57.041810+0000","last_fullsized":"2023-11-24T06:08:57.041810+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.041089+0000","last_change":"2023-11-24T06:08:57.041089+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.041089+0000","last_undegraded":"2023-11-24T06:08:57.041089+0000","last_fullsized":"2023-11-24T06:08:57.041089+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.042679+0000","last_change":"2023-11-24T06:08:57.042679+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.042679+0000","last_undegraded":"2023-11-24T06:08:57.042679+0000","last_fullsized":"2023-11-24T06:08:57.042679+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.039371+0000","last_change":"2023-11-24T06:08:57.039371+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.039371+0000","last_undegraded":"2023-11-24T06:08:57.039371+0000","last_fullsized":"2023-11-24T06:08:57.039371+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.038993+0000","last_change":"2023-11-24T06:08:57.038993+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.038993+0000","last_undegraded":"2023-11-24T06:08:57.038993+0000","last_fullsized":"2023-11-24T06:08:57.038993+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.039470+0000","last_change":"2023-11-24T06:08:57.039470+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.039470+0000","last_undegraded":"2023-11-24T06:08:57.039470+0000","last_fullsized":"2023-11-24T06:08:57.039470+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.037191+0000","last_change":"2023-11-24T06:08:57.037191+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.037191+0000","last_undegraded":"2023-11-24T06:08:57.037191+0000","last_fullsized":"2023-11-24T06:08:57.037191+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.036667+0000","last_change":"2023-11-24T06:08:57.036667+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.036667+0000","last_undegraded":"2023-11-24T06:08:57.036667+0000","last_fullsized":"2023-11-24T06:08:57.036667+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.039564+0000","last_change":"2023-11-24T06:08:57.039564+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.039564+0000","last_undegraded":"2023-11-24T06:08:57.039564+0000","last_fullsized":"2023-11-24T06:08:57.039564+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.038756+0000","last_change":"2023-11-24T06:08:57.038756+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.038756+0000","last_undegraded":"2023-11-24T06:08:57.038756+0000","last_fullsized":"2023-11-24T06:08:57.038756+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.040079+0000","last_change":"2023-11-24T06:08:57.040079+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.040079+0000","last_undegraded":"2023-11-24T06:08:57.040079+0000","last_fullsized":"2023-11-24T06:08:57.040079+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.036673+0000","last_change":"2023-11-24T06:08:57.036673+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.036673+0000","last_undegraded":"2023-11-24T06:08:57.036673+0000","last_fullsized":"2023-11-24T06:08:57.036673+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.039988+0000","last_change":"2023-11-24T06:08:57.039988+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.039988+0000","last_undegraded":"2023-11-24T06:08:57.039988+0000","last_fullsized":"2023-11-24T06:08:57.039988+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.036633+0000","last_change":"2023-11-24T06:08:57.036633+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.036633+0000","last_undegraded":"2023-11-24T06:08:57.036633+0000","last_fullsized":"2023-11-24T06:08:57.036633+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.040525+0000","last_change":"2023-11-24T06:08:57.040525+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.040525+0000","last_undegraded":"2023-11-24T06:08:57.040525+0000","last_fullsized":"2023-11-24T06:08:57.040525+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.039055+0000","last_change":"2023-11-24T06:08:57.039055+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.039055+0000","last_undegraded":"2023-11-24T06:08:57.039055+0000","last_fullsized":"2023-11-24T06:08:57.039055+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.040620+0000","last_change":"2023-11-24T06:08:57.040620+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.040620+0000","last_undegraded":"2023-11-24T06:08:57.040620+0000","last_fullsized":"2023-11-24T06:08:57.040620+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-11-24T06:08:58.034363+0000","last_change":"2023-11-24T06:08:58.034363+0000","last_active":"2023-11-24T06:08:58.034363+0000","last_peered":"2023-11-24T06:08:58.034363+0000","last_clean":"2023-11-24T06:08:58.034363+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:58.034363+0000","last_undegraded":"2023-11-24T06:08:58.034363+0000","last_fullsized":"2023-11-24T06:08:58.034363+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2023-11-24T06:08:57.036698+0000","last_change":"2023-11-24T06:08:57.036698+0000","last_active":"2023-11-24T06:08:57.028754+0000","last_peered":"2023-11-24T06:08:57.028754+0000","last_clean":"2023-11-24T06:08:57.028754+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-24T06:08:57.036698+0000","last_undegraded":"2023-11-24T06:08:57.036698+0000","last_fullsized":"2023-11-24T06:08:57.036698+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":174,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-11-24T06:08:52.067545+0000","last_change":"2023-11-24T06:07:49.463719+0000","last_active":"2023-11-24T06:08:52.067545+0000","last_peered":"2023-11-24T06:08:52.067545+0000","last_clean":"2023-11-24T06:08:52.067545+0000","last_became_active":"2023-11-24T06:07:49.462736+0000","last_became_peered":"2023-11-24T06:07:49.462736+0000","last_unstale":"2023-11-24T06:08:52.067545+0000","last_undegraded":"2023-11-24T06:08:52.067545+0000","last_fullsized":"2023-11-24T06:08:52.067545+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_clean_scrub_stamp":"2023-11-24T06:07:47.860604+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:16.508770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":717116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.52100000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.39900000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.47599999999999998}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.433},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.5}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.61499999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.57199999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.60499999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":126748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.66700000000000004}]},{"osd":1,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.77100000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.626}]},{"osd":3,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.753}]},{"osd":4,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.57699999999999996}]},{"osd":7,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.64100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":126748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.7670000000000003,"5min":4.7670000000000003,"15min":4.7670000000000003},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.0810000000000004,"5min":6.0810000000000004,"15min":6.0810000000000004},"last":0.39300000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.8340000000000001,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.45200000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.47599999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.9529999999999998,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.64800000000000002}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.9640000000000004,"5min":5.9640000000000004,"15min":5.9640000000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":3.02,"5min":3.02,"15min":3.02},"last":0.68500000000000005}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":6.048,"5min":6.048,"15min":6.048},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.9929999999999999,"5min":2.9929999999999999,"15min":2.9929999999999999},"last":0.61299999999999999}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":6.0129999999999999,"5min":6.0129999999999999,"15min":6.0129999999999999},"last":0.67000000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":929792,"data_stored":717116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":12.076000000000001,"5min":12.076000000000001,"15min":12.076000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":4.181,"5min":4.181,"15min":4.181},"last":0.47199999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.2089999999999996,"5min":4.2089999999999996,"15min":4.2089999999999996},"last":0.42299999999999999}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.439},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.45100000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":4.1520000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.63400000000000001}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":4.1260000000000003,"5min":4.1260000000000003,"15min":4.1260000000000003},"last":0.56399999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.67900000000000005}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.59599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":126748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.7379999999999995,"5min":9.7379999999999995,"15min":9.7379999999999995},"last":0.32900000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":11.967000000000001,"5min":11.967000000000001,"15min":11.967000000000001},"last":0.38},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":9.593,"5min":9.593,"15min":9.593},"last":0.40400000000000003}]},{"osd":3,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.34799999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":9.6479999999999997,"5min":9.6479999999999997,"15min":9.6479999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.499}]},{"osd":5,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":0.47299999999999998},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":9.7119999999999997,"5min":9.7119999999999997,"15min":9.7119999999999997},"last":0.58299999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.56599999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":9.6859999999999999,"5min":9.6859999999999999,"15min":9.6859999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.623}]}]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.34699999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.39500000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.36699999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.61499999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.53700000000000003}]},{"osd":6,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.63},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.59899999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.58899999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":126748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.9539999999999997,"5min":6.9539999999999997,"15min":6.9539999999999997},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":7.0039999999999996,"5min":7.0039999999999996,"15min":7.0039999999999996},"last":0.63}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":12.907,"5min":12.907,"15min":12.907},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":6.9710000000000001,"5min":6.9710000000000001,"15min":6.9710000000000001},"last":0.65100000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.59599999999999997}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":7.0540000000000003,"5min":7.0540000000000003,"15min":7.0540000000000003},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":7.0679999999999996,"5min":7.0679999999999996,"15min":7.0679999999999996},"last":0.61299999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.49399999999999999}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.53700000000000003}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.51800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":4.048},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":4.0090000000000003}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":3.992},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.45700000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.996,"5min":1.996,"15min":1.996},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.48999999999999999}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":4.0279999999999996}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.53100000000000003}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.441},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.50600000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.52000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:09:03.173 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:09:03.329 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.327+0000 7f0cb5600700 1 Processor -- start 2023-11-24T06:09:03.330 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.327+0000 7f0cb5600700 1 -- start start 2023-11-24T06:09:03.333 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.327+0000 7f0cb5600700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0cb015e350 0x7f0cb006f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:03.333 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.327+0000 7f0cb5600700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb0070080 con 0x7f0cb006fc30 2023-11-24T06:09:03.333 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.327+0000 7f0cb5600700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb00d10f0 con 0x7f0cb015df70 2023-11-24T06:09:03.333 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.327+0000 7f0cb5600700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb00d0850 con 0x7f0cb015e350 2023-11-24T06:09:03.334 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0caf7fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f0cb006fc30 0x7f0cb00ccaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54660/0 (socket says 172.21.15.91:54660) 2023-11-24T06:09:03.334 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0caf7fe700 1 -- 172.21.15.91:0/21679245 learned_addr learned my addr 172.21.15.91:0/21679245 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:03.334 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cae7fc700 1 --2- 172.21.15.91:0/21679245 >> v2:172.21.15.94:3300/0 conn(0x7f0cb015e350 0x7f0cb006f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:03.334 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cadffb700 1 -- 172.21.15.91:0/21679245 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712209618 0 0) 0x7f0ca8001490 con 0x7f0cb006fc30 2023-11-24T06:09:03.334 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cadffb700 1 -- 172.21.15.91:0/21679245 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c9c0035b0 con 0x7f0cb006fc30 2023-11-24T06:09:03.334 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cadffb700 1 -- 172.21.15.91:0/21679245 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 84472445 0 0) 0x7f0c9c0035b0 con 0x7f0cb006fc30 2023-11-24T06:09:03.334 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cadffb700 1 -- 172.21.15.91:0/21679245 >> v2:172.21.15.94:3300/0 conn(0x7f0cb015e350 msgr2=0x7f0cb006f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:03.334 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cadffb700 1 --2- 172.21.15.91:0/21679245 >> v2:172.21.15.94:3300/0 conn(0x7f0cb015e350 0x7f0cb006f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.334 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cadffb700 1 -- 172.21.15.91:0/21679245 >> v1:172.21.15.94:6789/0 conn(0x7f0cb015df70 legacy=0x7f0cb006eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:03.334 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cadffb700 1 -- 172.21.15.91:0/21679245 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb00d0b30 con 0x7f0cb006fc30 2023-11-24T06:09:03.334 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cadffb700 1 -- 172.21.15.91:0/21679245 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f0ca8001b80 con 0x7f0cb006fc30 2023-11-24T06:09:03.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cae7fc700 1 --2- 172.21.15.91:0/21679245 >> v2:172.21.15.94:3300/0 conn(0x7f0cb015e350 0x7f0cb006f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:09:03.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cadffb700 1 -- 172.21.15.91:0/21679245 >> v1:172.21.15.91:6789/0 conn(0x7f0cb006fc30 legacy=0x7f0cb00ccaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:03.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cadffb700 1 --2- 172.21.15.91:0/21679245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c9c003a00 0x7f0c9c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:03.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cadffb700 1 --2- 172.21.15.91:0/21679245 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c9c004850 0x7f0c9c004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:03.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cadffb700 1 --2- 172.21.15.91:0/21679245 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0c9c005210 0x7f0c9c00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:03.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cadffb700 1 -- 172.21.15.91:0/21679245 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb00d0850 con 0x7f0c9c005210 2023-11-24T06:09:03.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cadffb700 1 -- 172.21.15.91:0/21679245 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c00c0b0 con 0x7f0c9c003a00 2023-11-24T06:09:03.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cadffb700 1 -- 172.21.15.91:0/21679245 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c00c270 con 0x7f0c9c004850 2023-11-24T06:09:03.335 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0caf7fe700 1 --2- 172.21.15.91:0/21679245 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0c9c005210 0x7f0c9c00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:03.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0caeffd700 1 --2- 172.21.15.91:0/21679245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c9c003a00 0x7f0c9c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:03.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0cae7fc700 1 --2- 172.21.15.91:0/21679245 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c9c004850 0x7f0c9c004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:03.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0caf7fe700 1 -- 172.21.15.91:0/21679245 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c9c004850 msgr2=0x7f0c9c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:03.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.328+0000 7f0caf7fe700 1 --2- 172.21.15.91:0/21679245 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c9c004850 0x7f0c9c004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.329+0000 7f0caf7fe700 1 -- 172.21.15.91:0/21679245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c9c003a00 msgr2=0x7f0c9c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:03.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.329+0000 7f0caf7fe700 1 --2- 172.21.15.91:0/21679245 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0c9c003a00 0x7f0c9c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.329+0000 7f0caf7fe700 1 -- 172.21.15.91:0/21679245 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9c0035b0 con 0x7f0c9c005210 2023-11-24T06:09:03.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.329+0000 7f0caf7fe700 1 -- 172.21.15.91:0/21679245 reap_dead start 2023-11-24T06:09:03.336 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.329+0000 7f0caf7fe700 1 --2- 172.21.15.91:0/21679245 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0c9c005210 0x7f0c9c00ba10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800a830 tx=0x7f0c9c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5455015fe047e2da server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:03.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.329+0000 7f0cadffb700 1 -- 172.21.15.91:0/21679245 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca800a490 con 0x7f0c9c005210 2023-11-24T06:09:03.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.329+0000 7f0cadffb700 1 -- 172.21.15.91:0/21679245 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca8012620 con 0x7f0c9c005210 2023-11-24T06:09:03.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.329+0000 7f0cadffb700 1 -- 172.21.15.91:0/21679245 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca80129d0 con 0x7f0c9c005210 2023-11-24T06:09:03.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.329+0000 7f0cb5600700 1 -- 172.21.15.91:0/21679245 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0c9c005210 msgr2=0x7f0c9c00ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:03.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.329+0000 7f0cb5600700 1 --2- 172.21.15.91:0/21679245 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0c9c005210 0x7f0c9c00ba10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800a830 tx=0x7f0c9c005880 comp rx=0 tx=0).stop 2023-11-24T06:09:03.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.329+0000 7f0cb5600700 1 -- 172.21.15.91:0/21679245 shutdown_connections 2023-11-24T06:09:03.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.329+0000 7f0cb5600700 1 --2- 172.21.15.91:0/21679245 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0c9c005210 0x7f0c9c00ba10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.329+0000 7f0cb5600700 1 -- 172.21.15.91:0/21679245 >> 172.21.15.91:0/21679245 conn(0x7f0cb00b3720 msgr2=0x7f0cb00b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:03.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.329+0000 7f0cb5600700 1 -- 172.21.15.91:0/21679245 shutdown_connections 2023-11-24T06:09:03.337 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.329+0000 7f0cb5600700 1 -- 172.21.15.91:0/21679245 wait complete. 2023-11-24T06:09:03.340 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.331+0000 7f0cb5600700 1 Processor -- start 2023-11-24T06:09:03.340 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.331+0000 7f0cb5600700 1 -- start start 2023-11-24T06:09:03.340 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cb5600700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c9c005210 0x7f0cb015b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:03.340 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cb5600700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0cb00c7b90 0x7f0cb00c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:03.340 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cb5600700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cb006c930 0x7f0cb006cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:03.340 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cb5600700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb006d2f0 con 0x7f0cb00c7b90 2023-11-24T06:09:03.340 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cb5600700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb00b4ee0 con 0x7f0cb006c930 2023-11-24T06:09:03.340 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cb5600700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb00c7890 con 0x7f0c9c005210 2023-11-24T06:09:03.340 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cae7fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0cb00c7b90 0x7f0cb00c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:03.340 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cae7fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0cb00c7b90 0x7f0cb00c66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59736/0 (socket says 172.21.15.91:59736) 2023-11-24T06:09:03.340 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cae7fc700 1 -- 172.21.15.91:0/316892874 learned_addr learned my addr 172.21.15.91:0/316892874 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:03.341 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cae7fc700 1 -- 172.21.15.91:0/316892874 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c9c005210 msgr2=0x7f0cb015b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:03.341 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0caf7fe700 1 --2- 172.21.15.91:0/316892874 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cb006c930 0x7f0cb006cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:03.341 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cae7fc700 1 --2- 172.21.15.91:0/316892874 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c9c005210 0x7f0cb015b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.341 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cae7fc700 1 -- 172.21.15.91:0/316892874 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cb006c930 msgr2=0x7f0cb006cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:03.341 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cae7fc700 1 --2- 172.21.15.91:0/316892874 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cb006c930 0x7f0cb006cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.341 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cae7fc700 1 -- 172.21.15.91:0/316892874 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb00b51a0 con 0x7f0cb00c7b90 2023-11-24T06:09:03.341 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cae7fc700 1 --2- 172.21.15.91:0/316892874 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0cb00c7b90 0x7f0cb00c66f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0011ca0 tx=0x7f0ca000f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:03.341 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0ca77fe700 1 -- 172.21.15.91:0/316892874 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca000fad0 con 0x7f0cb00c7b90 2023-11-24T06:09:03.341 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0ca77fe700 1 -- 172.21.15.91:0/316892874 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca000fc30 con 0x7f0cb00c7b90 2023-11-24T06:09:03.341 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cb5600700 1 -- 172.21.15.91:0/316892874 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb015b920 con 0x7f0cb00c7b90 2023-11-24T06:09:03.341 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0ca77fe700 1 -- 172.21.15.91:0/316892874 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0004020 con 0x7f0cb00c7b90 2023-11-24T06:09:03.341 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.332+0000 7f0cb5600700 1 -- 172.21.15.91:0/316892874 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb00c72d0 con 0x7f0cb00c7b90 2023-11-24T06:09:03.342 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.333+0000 7f0cb5600700 1 -- 172.21.15.91:0/316892874 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb0067f60 con 0x7f0cb00c7b90 2023-11-24T06:09:03.342 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.334+0000 7f0ca77fe700 1 -- 172.21.15.91:0/316892874 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0ca0004180 con 0x7f0cb00c7b90 2023-11-24T06:09:03.342 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.334+0000 7f0ca77fe700 1 --2- 172.21.15.91:0/316892874 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0c9006aed0 0x7f0c9006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:03.342 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.334+0000 7f0ca77fe700 1 -- 172.21.15.91:0/316892874 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ca0089b80 con 0x7f0cb00c7b90 2023-11-24T06:09:03.342 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.334+0000 7f0caeffd700 1 --2- 172.21.15.91:0/316892874 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0c9006aed0 0x7f0c9006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:03.342 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.335+0000 7f0caeffd700 1 --2- 172.21.15.91:0/316892874 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0c9006aed0 0x7f0c9006d390 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0c98000c00 tx=0x7f0c98009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:03.342 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.339+0000 7f0ca77fe700 1 -- 172.21.15.91:0/316892874 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ca0054c30 con 0x7f0cb00c7b90 2023-11-24T06:09:03.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.527+0000 7f0cb5600700 1 -- 172.21.15.91:0/316892874 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0cb00cc580 con 0x7f0c9006aed0 2023-11-24T06:09:03.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.535+0000 7f0ca77fe700 1 -- 172.21.15.91:0/316892874 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288780 (secure 0 0 0) 0x7f0c98014450 con 0x7f0c9006aed0 2023-11-24T06:09:03.541 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:03.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.538+0000 7f0cb5600700 1 -- 172.21.15.91:0/316892874 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0c9006aed0 msgr2=0x7f0c9006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:03.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.538+0000 7f0cb5600700 1 --2- 172.21.15.91:0/316892874 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0c9006aed0 0x7f0c9006d390 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0c98000c00 tx=0x7f0c98009040 comp rx=0 tx=0).stop 2023-11-24T06:09:03.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.538+0000 7f0cb5600700 1 -- 172.21.15.91:0/316892874 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0cb00c7b90 msgr2=0x7f0cb00c66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:03.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.538+0000 7f0cb5600700 1 --2- 172.21.15.91:0/316892874 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0cb00c7b90 0x7f0cb00c66f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0011ca0 tx=0x7f0ca000f520 comp rx=0 tx=0).stop 2023-11-24T06:09:03.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.538+0000 7f0cb5600700 1 -- 172.21.15.91:0/316892874 shutdown_connections 2023-11-24T06:09:03.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.538+0000 7f0cb5600700 1 --2- 172.21.15.91:0/316892874 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0c9006aed0 0x7f0c9006d390 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.538+0000 7f0cb5600700 1 --2- 172.21.15.91:0/316892874 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0cb006c930 0x7f0cb006cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.538+0000 7f0cb5600700 1 --2- 172.21.15.91:0/316892874 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0c9c005210 0x7f0cb015b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.538+0000 7f0cb5600700 1 --2- 172.21.15.91:0/316892874 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0cb00c7b90 0x7f0cb00c66f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.538+0000 7f0cb5600700 1 -- 172.21.15.91:0/316892874 >> 172.21.15.91:0/316892874 conn(0x7f0cb00b3720 msgr2=0x7f0cb006c130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:03.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.538+0000 7f0cb5600700 1 -- 172.21.15.91:0/316892874 shutdown_connections 2023-11-24T06:09:03.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.538+0000 7f0cb5600700 1 -- 172.21.15.91:0/316892874 wait complete. 2023-11-24T06:09:03.544 INFO:teuthology.orchestra.run.smithi091.stderr:dumped all 2023-11-24T06:09:03.553 INFO:teuthology.orchestra.run.smithi091.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2023-11-24T06:09:03.409635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":165,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220448,"kb_used_data":4232,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754272,"statfs":{"total":773094113280,"available":772868374528,"internally_reserved":0,"allocated":4333568,"data_stored":2400812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001056"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047464+0000","last_change":"2023-11-24T06:08:59.067825+0000","last_active":"2023-11-24T06:09:00.047464+0000","last_peered":"2023-11-24T06:09:00.047464+0000","last_clean":"2023-11-24T06:09:00.047464+0000","last_became_active":"2023-11-24T06:08:59.067562+0000","last_became_peered":"2023-11-24T06:08:59.067562+0000","last_unstale":"2023-11-24T06:09:00.047464+0000","last_undegraded":"2023-11-24T06:09:00.047464+0000","last_fullsized":"2023-11-24T06:09:00.047464+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:06:05.979665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.048940+0000","last_change":"2023-11-24T06:08:59.074736+0000","last_active":"2023-11-24T06:09:00.048940+0000","last_peered":"2023-11-24T06:09:00.048940+0000","last_clean":"2023-11-24T06:09:00.048940+0000","last_became_active":"2023-11-24T06:08:59.074428+0000","last_became_peered":"2023-11-24T06:08:59.074428+0000","last_unstale":"2023-11-24T06:09:00.048940+0000","last_undegraded":"2023-11-24T06:09:00.048940+0000","last_fullsized":"2023-11-24T06:09:00.048940+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T14:47:09.630954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.394769+0000","last_change":"2023-11-24T06:08:59.075775+0000","last_active":"2023-11-24T06:09:00.394769+0000","last_peered":"2023-11-24T06:09:00.394769+0000","last_clean":"2023-11-24T06:09:00.394769+0000","last_became_active":"2023-11-24T06:08:59.075475+0000","last_became_peered":"2023-11-24T06:08:59.075475+0000","last_unstale":"2023-11-24T06:09:00.394769+0000","last_undegraded":"2023-11-24T06:09:00.394769+0000","last_fullsized":"2023-11-24T06:09:00.394769+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:57.493230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.129804+0000","last_change":"2023-11-24T06:08:59.062182+0000","last_active":"2023-11-24T06:09:00.129804+0000","last_peered":"2023-11-24T06:09:00.129804+0000","last_clean":"2023-11-24T06:09:00.129804+0000","last_became_active":"2023-11-24T06:08:59.061907+0000","last_became_peered":"2023-11-24T06:08:59.061907+0000","last_unstale":"2023-11-24T06:09:00.129804+0000","last_undegraded":"2023-11-24T06:09:00.129804+0000","last_fullsized":"2023-11-24T06:09:00.129804+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:52:22.319013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.393714+0000","last_change":"2023-11-24T06:08:59.063842+0000","last_active":"2023-11-24T06:09:00.393714+0000","last_peered":"2023-11-24T06:09:00.393714+0000","last_clean":"2023-11-24T06:09:00.393714+0000","last_became_active":"2023-11-24T06:08:59.063575+0000","last_became_peered":"2023-11-24T06:08:59.063575+0000","last_unstale":"2023-11-24T06:09:00.393714+0000","last_undegraded":"2023-11-24T06:09:00.393714+0000","last_fullsized":"2023-11-24T06:09:00.393714+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:49:58.642923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.048408+0000","last_change":"2023-11-24T06:08:59.072737+0000","last_active":"2023-11-24T06:09:00.048408+0000","last_peered":"2023-11-24T06:09:00.048408+0000","last_clean":"2023-11-24T06:09:00.048408+0000","last_became_active":"2023-11-24T06:08:59.069749+0000","last_became_peered":"2023-11-24T06:08:59.069749+0000","last_unstale":"2023-11-24T06:09:00.048408+0000","last_undegraded":"2023-11-24T06:09:00.048408+0000","last_fullsized":"2023-11-24T06:09:00.048408+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:24:51.864974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047334+0000","last_change":"2023-11-24T06:08:59.062847+0000","last_active":"2023-11-24T06:09:00.047334+0000","last_peered":"2023-11-24T06:09:00.047334+0000","last_clean":"2023-11-24T06:09:00.047334+0000","last_became_active":"2023-11-24T06:08:59.062297+0000","last_became_peered":"2023-11-24T06:08:59.062297+0000","last_unstale":"2023-11-24T06:09:00.047334+0000","last_undegraded":"2023-11-24T06:09:00.047334+0000","last_fullsized":"2023-11-24T06:09:00.047334+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:52:22.319013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.046742+0000","last_change":"2023-11-24T06:08:59.068080+0000","last_active":"2023-11-24T06:09:00.046742+0000","last_peered":"2023-11-24T06:09:00.046742+0000","last_clean":"2023-11-24T06:09:00.046742+0000","last_became_active":"2023-11-24T06:08:59.067642+0000","last_became_peered":"2023-11-24T06:08:59.067642+0000","last_unstale":"2023-11-24T06:09:00.046742+0000","last_undegraded":"2023-11-24T06:09:00.046742+0000","last_fullsized":"2023-11-24T06:09:00.046742+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:29.210131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.395257+0000","last_change":"2023-11-24T06:08:59.068899+0000","last_active":"2023-11-24T06:09:00.395257+0000","last_peered":"2023-11-24T06:09:00.395257+0000","last_clean":"2023-11-24T06:09:00.395257+0000","last_became_active":"2023-11-24T06:08:59.068466+0000","last_became_peered":"2023-11-24T06:08:59.068466+0000","last_unstale":"2023-11-24T06:09:00.395257+0000","last_undegraded":"2023-11-24T06:09:00.395257+0000","last_fullsized":"2023-11-24T06:09:00.395257+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:09:48.969762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.129976+0000","last_change":"2023-11-24T06:08:59.075607+0000","last_active":"2023-11-24T06:09:00.129976+0000","last_peered":"2023-11-24T06:09:00.129976+0000","last_clean":"2023-11-24T06:09:00.129976+0000","last_became_active":"2023-11-24T06:08:59.075267+0000","last_became_peered":"2023-11-24T06:08:59.075267+0000","last_unstale":"2023-11-24T06:09:00.129976+0000","last_undegraded":"2023-11-24T06:09:00.129976+0000","last_fullsized":"2023-11-24T06:09:00.129976+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:37:54.962724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047638+0000","last_change":"2023-11-24T06:08:59.068456+0000","last_active":"2023-11-24T06:09:00.047638+0000","last_peered":"2023-11-24T06:09:00.047638+0000","last_clean":"2023-11-24T06:09:00.047638+0000","last_became_active":"2023-11-24T06:08:59.068200+0000","last_became_peered":"2023-11-24T06:08:59.068200+0000","last_unstale":"2023-11-24T06:09:00.047638+0000","last_undegraded":"2023-11-24T06:09:00.047638+0000","last_fullsized":"2023-11-24T06:09:00.047638+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:24:51.864974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.394581+0000","last_change":"2023-11-24T06:08:59.077347+0000","last_active":"2023-11-24T06:09:00.394581+0000","last_peered":"2023-11-24T06:09:00.394581+0000","last_clean":"2023-11-24T06:09:00.394581+0000","last_became_active":"2023-11-24T06:08:59.077044+0000","last_became_peered":"2023-11-24T06:08:59.077044+0000","last_unstale":"2023-11-24T06:09:00.394581+0000","last_undegraded":"2023-11-24T06:09:00.394581+0000","last_fullsized":"2023-11-24T06:09:00.394581+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:37:54.962724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.393852+0000","last_change":"2023-11-24T06:08:59.064581+0000","last_active":"2023-11-24T06:09:00.393852+0000","last_peered":"2023-11-24T06:09:00.393852+0000","last_clean":"2023-11-24T06:09:00.393852+0000","last_became_active":"2023-11-24T06:08:59.064287+0000","last_became_peered":"2023-11-24T06:08:59.064287+0000","last_unstale":"2023-11-24T06:09:00.393852+0000","last_undegraded":"2023-11-24T06:09:00.393852+0000","last_fullsized":"2023-11-24T06:09:00.393852+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:55:32.423643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.048522+0000","last_change":"2023-11-24T06:08:59.067921+0000","last_active":"2023-11-24T06:09:00.048522+0000","last_peered":"2023-11-24T06:09:00.048522+0000","last_clean":"2023-11-24T06:09:00.048522+0000","last_became_active":"2023-11-24T06:08:59.067550+0000","last_became_peered":"2023-11-24T06:08:59.067550+0000","last_unstale":"2023-11-24T06:09:00.048522+0000","last_undegraded":"2023-11-24T06:09:00.048522+0000","last_fullsized":"2023-11-24T06:09:00.048522+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:44:29.272662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.394869+0000","last_change":"2023-11-24T06:08:59.067482+0000","last_active":"2023-11-24T06:09:00.394869+0000","last_peered":"2023-11-24T06:09:00.394869+0000","last_clean":"2023-11-24T06:09:00.394869+0000","last_became_active":"2023-11-24T06:08:59.067196+0000","last_became_peered":"2023-11-24T06:08:59.067196+0000","last_unstale":"2023-11-24T06:09:00.394869+0000","last_undegraded":"2023-11-24T06:09:00.394869+0000","last_fullsized":"2023-11-24T06:09:00.394869+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:56:24.331211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047200+0000","last_change":"2023-11-24T06:08:59.077309+0000","last_active":"2023-11-24T06:09:00.047200+0000","last_peered":"2023-11-24T06:09:00.047200+0000","last_clean":"2023-11-24T06:09:00.047200+0000","last_became_active":"2023-11-24T06:08:59.076712+0000","last_became_peered":"2023-11-24T06:08:59.076712+0000","last_unstale":"2023-11-24T06:09:00.047200+0000","last_undegraded":"2023-11-24T06:09:00.047200+0000","last_fullsized":"2023-11-24T06:09:00.047200+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:56:24.331211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047759+0000","last_change":"2023-11-24T06:08:59.061088+0000","last_active":"2023-11-24T06:09:00.047759+0000","last_peered":"2023-11-24T06:09:00.047759+0000","last_clean":"2023-11-24T06:09:00.047759+0000","last_became_active":"2023-11-24T06:08:59.059470+0000","last_became_peered":"2023-11-24T06:08:59.059470+0000","last_unstale":"2023-11-24T06:09:00.047759+0000","last_undegraded":"2023-11-24T06:09:00.047759+0000","last_fullsized":"2023-11-24T06:09:00.047759+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:44:29.272662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.049005+0000","last_change":"2023-11-24T06:08:59.073252+0000","last_active":"2023-11-24T06:09:00.049005+0000","last_peered":"2023-11-24T06:09:00.049005+0000","last_clean":"2023-11-24T06:09:00.049005+0000","last_became_active":"2023-11-24T06:08:59.071185+0000","last_became_peered":"2023-11-24T06:08:59.071185+0000","last_unstale":"2023-11-24T06:09:00.049005+0000","last_undegraded":"2023-11-24T06:09:00.049005+0000","last_fullsized":"2023-11-24T06:09:00.049005+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:19:13.117043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.048828+0000","last_change":"2023-11-24T06:08:59.068237+0000","last_active":"2023-11-24T06:09:00.048828+0000","last_peered":"2023-11-24T06:09:00.048828+0000","last_clean":"2023-11-24T06:09:00.048828+0000","last_became_active":"2023-11-24T06:08:59.067970+0000","last_became_peered":"2023-11-24T06:08:59.067970+0000","last_unstale":"2023-11-24T06:09:00.048828+0000","last_undegraded":"2023-11-24T06:09:00.048828+0000","last_fullsized":"2023-11-24T06:09:00.048828+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:16:21.736518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.393820+0000","last_change":"2023-11-24T06:08:59.055091+0000","last_active":"2023-11-24T06:09:00.393820+0000","last_peered":"2023-11-24T06:09:00.393820+0000","last_clean":"2023-11-24T06:09:00.393820+0000","last_became_active":"2023-11-24T06:08:59.054935+0000","last_became_peered":"2023-11-24T06:08:59.054935+0000","last_unstale":"2023-11-24T06:09:00.393820+0000","last_undegraded":"2023-11-24T06:09:00.393820+0000","last_fullsized":"2023-11-24T06:09:00.393820+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:37:10.309849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.394699+0000","last_change":"2023-11-24T06:08:59.064106+0000","last_active":"2023-11-24T06:09:00.394699+0000","last_peered":"2023-11-24T06:09:00.394699+0000","last_clean":"2023-11-24T06:09:00.394699+0000","last_became_active":"2023-11-24T06:08:59.063849+0000","last_became_peered":"2023-11-24T06:08:59.063849+0000","last_unstale":"2023-11-24T06:09:00.394699+0000","last_undegraded":"2023-11-24T06:09:00.394699+0000","last_fullsized":"2023-11-24T06:09:00.394699+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:52:22.319013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.395206+0000","last_change":"2023-11-24T06:08:59.063624+0000","last_active":"2023-11-24T06:09:00.395206+0000","last_peered":"2023-11-24T06:09:00.395206+0000","last_clean":"2023-11-24T06:09:00.395206+0000","last_became_active":"2023-11-24T06:08:59.063320+0000","last_became_peered":"2023-11-24T06:08:59.063320+0000","last_unstale":"2023-11-24T06:09:00.395206+0000","last_undegraded":"2023-11-24T06:09:00.395206+0000","last_fullsized":"2023-11-24T06:09:00.395206+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:10:42.190687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.127853+0000","last_change":"2023-11-24T06:08:59.062837+0000","last_active":"2023-11-24T06:09:00.127853+0000","last_peered":"2023-11-24T06:09:00.127853+0000","last_clean":"2023-11-24T06:09:00.127853+0000","last_became_active":"2023-11-24T06:08:59.062581+0000","last_became_peered":"2023-11-24T06:08:59.062581+0000","last_unstale":"2023-11-24T06:09:00.127853+0000","last_undegraded":"2023-11-24T06:09:00.127853+0000","last_fullsized":"2023-11-24T06:09:00.127853+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:42:13.536680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.394296+0000","last_change":"2023-11-24T06:08:59.058265+0000","last_active":"2023-11-24T06:09:00.394296+0000","last_peered":"2023-11-24T06:09:00.394296+0000","last_clean":"2023-11-24T06:09:00.394296+0000","last_became_active":"2023-11-24T06:08:59.058001+0000","last_became_peered":"2023-11-24T06:08:59.058001+0000","last_unstale":"2023-11-24T06:09:00.394296+0000","last_undegraded":"2023-11-24T06:09:00.394296+0000","last_fullsized":"2023-11-24T06:09:00.394296+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:29.210131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.130104+0000","last_change":"2023-11-24T06:08:59.066976+0000","last_active":"2023-11-24T06:09:00.130104+0000","last_peered":"2023-11-24T06:09:00.130104+0000","last_clean":"2023-11-24T06:09:00.130104+0000","last_became_active":"2023-11-24T06:08:59.066458+0000","last_became_peered":"2023-11-24T06:08:59.066458+0000","last_unstale":"2023-11-24T06:09:00.130104+0000","last_undegraded":"2023-11-24T06:09:00.130104+0000","last_fullsized":"2023-11-24T06:09:00.130104+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:57.493230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.049070+0000","last_change":"2023-11-24T06:08:59.073434+0000","last_active":"2023-11-24T06:09:00.049070+0000","last_peered":"2023-11-24T06:09:00.049070+0000","last_clean":"2023-11-24T06:09:00.049070+0000","last_became_active":"2023-11-24T06:08:59.072998+0000","last_became_peered":"2023-11-24T06:08:59.072998+0000","last_unstale":"2023-11-24T06:09:00.049070+0000","last_undegraded":"2023-11-24T06:09:00.049070+0000","last_fullsized":"2023-11-24T06:09:00.049070+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:09:44.000543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.130066+0000","last_change":"2023-11-24T06:08:59.067028+0000","last_active":"2023-11-24T06:09:00.130066+0000","last_peered":"2023-11-24T06:09:00.130066+0000","last_clean":"2023-11-24T06:09:00.130066+0000","last_became_active":"2023-11-24T06:08:59.062370+0000","last_became_peered":"2023-11-24T06:08:59.062370+0000","last_unstale":"2023-11-24T06:09:00.130066+0000","last_undegraded":"2023-11-24T06:09:00.130066+0000","last_fullsized":"2023-11-24T06:09:00.130066+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:09:48.969762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.394489+0000","last_change":"2023-11-24T06:08:59.060169+0000","last_active":"2023-11-24T06:09:00.394489+0000","last_peered":"2023-11-24T06:09:00.394489+0000","last_clean":"2023-11-24T06:09:00.394489+0000","last_became_active":"2023-11-24T06:08:59.059903+0000","last_became_peered":"2023-11-24T06:08:59.059903+0000","last_unstale":"2023-11-24T06:09:00.394489+0000","last_undegraded":"2023-11-24T06:09:00.394489+0000","last_fullsized":"2023-11-24T06:09:00.394489+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:52:22.319013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.394873+0000","last_change":"2023-11-24T06:08:59.063529+0000","last_active":"2023-11-24T06:09:00.394873+0000","last_peered":"2023-11-24T06:09:00.394873+0000","last_clean":"2023-11-24T06:09:00.394873+0000","last_became_active":"2023-11-24T06:08:59.063204+0000","last_became_peered":"2023-11-24T06:08:59.063204+0000","last_unstale":"2023-11-24T06:09:00.394873+0000","last_undegraded":"2023-11-24T06:09:00.394873+0000","last_fullsized":"2023-11-24T06:09:00.394873+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:16:02.211633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047423+0000","last_change":"2023-11-24T06:08:59.055128+0000","last_active":"2023-11-24T06:09:00.047423+0000","last_peered":"2023-11-24T06:09:00.047423+0000","last_clean":"2023-11-24T06:09:00.047423+0000","last_became_active":"2023-11-24T06:08:59.054851+0000","last_became_peered":"2023-11-24T06:08:59.054851+0000","last_unstale":"2023-11-24T06:09:00.047423+0000","last_undegraded":"2023-11-24T06:09:00.047423+0000","last_fullsized":"2023-11-24T06:09:00.047423+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:16:02.211633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047334+0000","last_change":"2023-11-24T06:08:59.055598+0000","last_active":"2023-11-24T06:09:00.047334+0000","last_peered":"2023-11-24T06:09:00.047334+0000","last_clean":"2023-11-24T06:09:00.047334+0000","last_became_active":"2023-11-24T06:08:59.055149+0000","last_became_peered":"2023-11-24T06:08:59.055149+0000","last_unstale":"2023-11-24T06:09:00.047334+0000","last_undegraded":"2023-11-24T06:09:00.047334+0000","last_fullsized":"2023-11-24T06:09:00.047334+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:42:13.536680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047519+0000","last_change":"2023-11-24T06:08:59.066793+0000","last_active":"2023-11-24T06:09:00.047519+0000","last_peered":"2023-11-24T06:09:00.047519+0000","last_clean":"2023-11-24T06:09:00.047519+0000","last_became_active":"2023-11-24T06:08:59.066282+0000","last_became_peered":"2023-11-24T06:08:59.066282+0000","last_unstale":"2023-11-24T06:09:00.047519+0000","last_undegraded":"2023-11-24T06:09:00.047519+0000","last_fullsized":"2023-11-24T06:09:00.047519+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:10:42.190687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.127918+0000","last_change":"2023-11-24T06:08:59.061464+0000","last_active":"2023-11-24T06:09:00.127918+0000","last_peered":"2023-11-24T06:09:00.127918+0000","last_clean":"2023-11-24T06:09:00.127918+0000","last_became_active":"2023-11-24T06:08:59.061171+0000","last_became_peered":"2023-11-24T06:08:59.061171+0000","last_unstale":"2023-11-24T06:09:00.127918+0000","last_undegraded":"2023-11-24T06:09:00.127918+0000","last_fullsized":"2023-11-24T06:09:00.127918+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:06:05.979665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.071641+0000","last_change":"2023-11-24T06:08:58.057234+0000","last_active":"2023-11-24T06:08:59.071641+0000","last_peered":"2023-11-24T06:08:59.071641+0000","last_clean":"2023-11-24T06:08:59.071641+0000","last_became_active":"2023-11-24T06:08:58.056839+0000","last_became_peered":"2023-11-24T06:08:58.056839+0000","last_unstale":"2023-11-24T06:08:59.071641+0000","last_undegraded":"2023-11-24T06:08:59.071641+0000","last_fullsized":"2023-11-24T06:08:59.071641+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:31.119037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047597+0000","last_change":"2023-11-24T06:08:59.060358+0000","last_active":"2023-11-24T06:09:00.047597+0000","last_peered":"2023-11-24T06:09:00.047597+0000","last_clean":"2023-11-24T06:09:00.047597+0000","last_became_active":"2023-11-24T06:08:59.060091+0000","last_became_peered":"2023-11-24T06:08:59.060091+0000","last_unstale":"2023-11-24T06:09:00.047597+0000","last_undegraded":"2023-11-24T06:09:00.047597+0000","last_fullsized":"2023-11-24T06:09:00.047597+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:42:15.629420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.063007+0000","last_change":"2023-11-24T06:08:58.059537+0000","last_active":"2023-11-24T06:08:59.063007+0000","last_peered":"2023-11-24T06:08:59.063007+0000","last_clean":"2023-11-24T06:08:59.063007+0000","last_became_active":"2023-11-24T06:08:58.059316+0000","last_became_peered":"2023-11-24T06:08:58.059316+0000","last_unstale":"2023-11-24T06:08:59.063007+0000","last_undegraded":"2023-11-24T06:08:59.063007+0000","last_fullsized":"2023-11-24T06:08:59.063007+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:14:10.378002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.394491+0000","last_change":"2023-11-24T06:08:59.074361+0000","last_active":"2023-11-24T06:09:00.394491+0000","last_peered":"2023-11-24T06:09:00.394491+0000","last_clean":"2023-11-24T06:09:00.394491+0000","last_became_active":"2023-11-24T06:08:59.074085+0000","last_became_peered":"2023-11-24T06:08:59.074085+0000","last_unstale":"2023-11-24T06:09:00.394491+0000","last_undegraded":"2023-11-24T06:09:00.394491+0000","last_fullsized":"2023-11-24T06:09:00.394491+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:29.210131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.075510+0000","last_change":"2023-11-24T06:08:58.063794+0000","last_active":"2023-11-24T06:08:59.075510+0000","last_peered":"2023-11-24T06:08:59.075510+0000","last_clean":"2023-11-24T06:08:59.075510+0000","last_became_active":"2023-11-24T06:08:58.063554+0000","last_became_peered":"2023-11-24T06:08:58.063554+0000","last_unstale":"2023-11-24T06:08:59.075510+0000","last_undegraded":"2023-11-24T06:08:59.075510+0000","last_fullsized":"2023-11-24T06:08:59.075510+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:52:21.313404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.130037+0000","last_change":"2023-11-24T06:08:59.062334+0000","last_active":"2023-11-24T06:09:00.130037+0000","last_peered":"2023-11-24T06:09:00.130037+0000","last_clean":"2023-11-24T06:09:00.130037+0000","last_became_active":"2023-11-24T06:08:59.062031+0000","last_became_peered":"2023-11-24T06:08:59.062031+0000","last_unstale":"2023-11-24T06:09:00.130037+0000","last_undegraded":"2023-11-24T06:09:00.130037+0000","last_fullsized":"2023-11-24T06:09:00.130037+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:10:42.190687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.067390+0000","last_change":"2023-11-24T06:08:58.067182+0000","last_active":"2023-11-24T06:08:59.067390+0000","last_peered":"2023-11-24T06:08:59.067390+0000","last_clean":"2023-11-24T06:08:59.067390+0000","last_became_active":"2023-11-24T06:08:58.066802+0000","last_became_peered":"2023-11-24T06:08:58.066802+0000","last_unstale":"2023-11-24T06:08:59.067390+0000","last_undegraded":"2023-11-24T06:08:59.067390+0000","last_fullsized":"2023-11-24T06:08:59.067390+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:40:49.492590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047609+0000","last_change":"2023-11-24T06:08:59.065952+0000","last_active":"2023-11-24T06:09:00.047609+0000","last_peered":"2023-11-24T06:09:00.047609+0000","last_clean":"2023-11-24T06:09:00.047609+0000","last_became_active":"2023-11-24T06:08:59.065591+0000","last_became_peered":"2023-11-24T06:08:59.065591+0000","last_unstale":"2023-11-24T06:09:00.047609+0000","last_undegraded":"2023-11-24T06:09:00.047609+0000","last_fullsized":"2023-11-24T06:09:00.047609+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:46:44.218249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.075937+0000","last_change":"2023-11-24T06:08:58.054118+0000","last_active":"2023-11-24T06:08:59.075937+0000","last_peered":"2023-11-24T06:08:59.075937+0000","last_clean":"2023-11-24T06:08:59.075937+0000","last_became_active":"2023-11-24T06:08:58.053841+0000","last_became_peered":"2023-11-24T06:08:58.053841+0000","last_unstale":"2023-11-24T06:08:59.075937+0000","last_undegraded":"2023-11-24T06:08:59.075937+0000","last_fullsized":"2023-11-24T06:08:59.075937+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:42:14.623811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.394525+0000","last_change":"2023-11-24T06:08:59.064095+0000","last_active":"2023-11-24T06:09:00.394525+0000","last_peered":"2023-11-24T06:09:00.394525+0000","last_clean":"2023-11-24T06:09:00.394525+0000","last_became_active":"2023-11-24T06:08:59.063816+0000","last_became_peered":"2023-11-24T06:08:59.063816+0000","last_unstale":"2023-11-24T06:09:00.394525+0000","last_undegraded":"2023-11-24T06:09:00.394525+0000","last_fullsized":"2023-11-24T06:09:00.394525+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:16:02.211633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.076220+0000","last_change":"2023-11-24T06:08:58.054287+0000","last_active":"2023-11-24T06:08:59.076220+0000","last_peered":"2023-11-24T06:08:59.076220+0000","last_clean":"2023-11-24T06:08:59.076220+0000","last_became_active":"2023-11-24T06:08:58.053977+0000","last_became_peered":"2023-11-24T06:08:58.053977+0000","last_unstale":"2023-11-24T06:08:59.076220+0000","last_undegraded":"2023-11-24T06:08:59.076220+0000","last_fullsized":"2023-11-24T06:08:59.076220+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:54:29.691305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.395098+0000","last_change":"2023-11-24T06:08:59.057129+0000","last_active":"2023-11-24T06:09:00.395098+0000","last_peered":"2023-11-24T06:09:00.395098+0000","last_clean":"2023-11-24T06:09:00.395098+0000","last_became_active":"2023-11-24T06:08:59.056862+0000","last_became_peered":"2023-11-24T06:08:59.056862+0000","last_unstale":"2023-11-24T06:09:00.395098+0000","last_undegraded":"2023-11-24T06:09:00.395098+0000","last_fullsized":"2023-11-24T06:09:00.395098+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:37:54.962724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.056920+0000","last_change":"2023-11-24T06:08:58.066589+0000","last_active":"2023-11-24T06:08:59.056920+0000","last_peered":"2023-11-24T06:08:59.056920+0000","last_clean":"2023-11-24T06:08:59.056920+0000","last_became_active":"2023-11-24T06:08:58.066273+0000","last_became_peered":"2023-11-24T06:08:58.066273+0000","last_unstale":"2023-11-24T06:08:59.056920+0000","last_undegraded":"2023-11-24T06:08:59.056920+0000","last_fullsized":"2023-11-24T06:08:59.056920+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:42:14.623811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.129632+0000","last_change":"2023-11-24T06:08:59.069247+0000","last_active":"2023-11-24T06:09:00.129632+0000","last_peered":"2023-11-24T06:09:00.129632+0000","last_clean":"2023-11-24T06:09:00.129632+0000","last_became_active":"2023-11-24T06:08:59.068979+0000","last_became_peered":"2023-11-24T06:08:59.068979+0000","last_unstale":"2023-11-24T06:09:00.129632+0000","last_undegraded":"2023-11-24T06:09:00.129632+0000","last_fullsized":"2023-11-24T06:09:00.129632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:56:24.331211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.051288+0000","last_change":"2023-11-24T06:08:58.052063+0000","last_active":"2023-11-24T06:08:59.051288+0000","last_peered":"2023-11-24T06:08:59.051288+0000","last_clean":"2023-11-24T06:08:59.051288+0000","last_became_active":"2023-11-24T06:08:58.051775+0000","last_became_peered":"2023-11-24T06:08:58.051775+0000","last_unstale":"2023-11-24T06:08:59.051288+0000","last_undegraded":"2023-11-24T06:08:59.051288+0000","last_fullsized":"2023-11-24T06:08:59.051288+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:55:31.418034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.394681+0000","last_change":"2023-11-24T06:08:59.077309+0000","last_active":"2023-11-24T06:09:00.394681+0000","last_peered":"2023-11-24T06:09:00.394681+0000","last_clean":"2023-11-24T06:09:00.394681+0000","last_became_active":"2023-11-24T06:08:59.077028+0000","last_became_peered":"2023-11-24T06:08:59.077028+0000","last_unstale":"2023-11-24T06:09:00.394681+0000","last_undegraded":"2023-11-24T06:09:00.394681+0000","last_fullsized":"2023-11-24T06:09:00.394681+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:10:42.190687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.057471+0000","last_change":"2023-11-24T06:08:58.060232+0000","last_active":"2023-11-24T06:08:59.057471+0000","last_peered":"2023-11-24T06:08:59.057471+0000","last_clean":"2023-11-24T06:08:59.057471+0000","last_became_active":"2023-11-24T06:08:58.059908+0000","last_became_peered":"2023-11-24T06:08:58.059908+0000","last_unstale":"2023-11-24T06:08:59.057471+0000","last_undegraded":"2023-11-24T06:08:59.057471+0000","last_fullsized":"2023-11-24T06:08:59.057471+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:16:01.206023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.049241+0000","last_change":"2023-11-24T06:08:59.070988+0000","last_active":"2023-11-24T06:09:00.049241+0000","last_peered":"2023-11-24T06:09:00.049241+0000","last_clean":"2023-11-24T06:09:00.049241+0000","last_became_active":"2023-11-24T06:08:59.070726+0000","last_became_peered":"2023-11-24T06:08:59.070726+0000","last_unstale":"2023-11-24T06:09:00.049241+0000","last_undegraded":"2023-11-24T06:09:00.049241+0000","last_fullsized":"2023-11-24T06:09:00.049241+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:28:53.742434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.050278+0000","last_change":"2023-11-24T06:08:58.054198+0000","last_active":"2023-11-24T06:08:59.050278+0000","last_peered":"2023-11-24T06:08:59.050278+0000","last_clean":"2023-11-24T06:08:59.050278+0000","last_became_active":"2023-11-24T06:08:58.053862+0000","last_became_peered":"2023-11-24T06:08:58.053862+0000","last_unstale":"2023-11-24T06:08:59.050278+0000","last_undegraded":"2023-11-24T06:08:59.050278+0000","last_fullsized":"2023-11-24T06:08:59.050278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:54:29.691305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.048600+0000","last_change":"2023-11-24T06:08:59.073464+0000","last_active":"2023-11-24T06:09:00.048600+0000","last_peered":"2023-11-24T06:09:00.048600+0000","last_clean":"2023-11-24T06:09:00.048600+0000","last_became_active":"2023-11-24T06:08:59.070071+0000","last_became_peered":"2023-11-24T06:08:59.070071+0000","last_unstale":"2023-11-24T06:09:00.048600+0000","last_undegraded":"2023-11-24T06:09:00.048600+0000","last_fullsized":"2023-11-24T06:09:00.048600+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:38:12.508483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.072238+0000","last_change":"2023-11-24T06:08:58.061691+0000","last_active":"2023-11-24T06:08:59.072238+0000","last_peered":"2023-11-24T06:08:59.072238+0000","last_clean":"2023-11-24T06:08:59.072238+0000","last_became_active":"2023-11-24T06:08:58.061401+0000","last_became_peered":"2023-11-24T06:08:58.061401+0000","last_unstale":"2023-11-24T06:08:59.072238+0000","last_undegraded":"2023-11-24T06:08:59.072238+0000","last_fullsized":"2023-11-24T06:08:59.072238+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:14:13.475472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.394909+0000","last_change":"2023-11-24T06:08:59.075669+0000","last_active":"2023-11-24T06:09:00.394909+0000","last_peered":"2023-11-24T06:09:00.394909+0000","last_clean":"2023-11-24T06:09:00.394909+0000","last_became_active":"2023-11-24T06:08:59.075392+0000","last_became_peered":"2023-11-24T06:08:59.075392+0000","last_unstale":"2023-11-24T06:09:00.394909+0000","last_undegraded":"2023-11-24T06:09:00.394909+0000","last_fullsized":"2023-11-24T06:09:00.394909+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:24:51.864974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.083557+0000","last_change":"2023-11-24T06:08:58.063728+0000","last_active":"2023-11-24T06:08:59.083557+0000","last_peered":"2023-11-24T06:08:59.083557+0000","last_clean":"2023-11-24T06:08:59.083557+0000","last_became_active":"2023-11-24T06:08:58.063453+0000","last_became_peered":"2023-11-24T06:08:58.063453+0000","last_unstale":"2023-11-24T06:08:59.083557+0000","last_undegraded":"2023-11-24T06:08:59.083557+0000","last_fullsized":"2023-11-24T06:08:59.083557+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:56.487621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.129605+0000","last_change":"2023-11-24T06:08:59.069577+0000","last_active":"2023-11-24T06:09:00.129605+0000","last_peered":"2023-11-24T06:09:00.129605+0000","last_clean":"2023-11-24T06:09:00.129605+0000","last_became_active":"2023-11-24T06:08:59.069331+0000","last_became_peered":"2023-11-24T06:08:59.069331+0000","last_unstale":"2023-11-24T06:09:00.129605+0000","last_undegraded":"2023-11-24T06:09:00.129605+0000","last_fullsized":"2023-11-24T06:09:00.129605+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:53:29.210131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.065268+0000","last_change":"2023-11-24T06:08:58.052717+0000","last_active":"2023-11-24T06:08:59.065268+0000","last_peered":"2023-11-24T06:08:59.065268+0000","last_clean":"2023-11-24T06:08:59.065268+0000","last_became_active":"2023-11-24T06:08:58.051598+0000","last_became_peered":"2023-11-24T06:08:58.051598+0000","last_unstale":"2023-11-24T06:08:59.065268+0000","last_undegraded":"2023-11-24T06:08:59.065268+0000","last_fullsized":"2023-11-24T06:08:59.065268+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:14:13.475472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.048551+0000","last_change":"2023-11-24T06:08:59.068689+0000","last_active":"2023-11-24T06:09:00.048551+0000","last_peered":"2023-11-24T06:09:00.048551+0000","last_clean":"2023-11-24T06:09:00.048551+0000","last_became_active":"2023-11-24T06:08:59.068291+0000","last_became_peered":"2023-11-24T06:08:59.068291+0000","last_unstale":"2023-11-24T06:09:00.048551+0000","last_undegraded":"2023-11-24T06:09:00.048551+0000","last_fullsized":"2023-11-24T06:09:00.048551+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:52:24.932072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.053202+0000","last_change":"2023-11-24T06:08:58.064183+0000","last_active":"2023-11-24T06:08:59.053202+0000","last_peered":"2023-11-24T06:08:59.053202+0000","last_clean":"2023-11-24T06:08:59.053202+0000","last_became_active":"2023-11-24T06:08:58.062992+0000","last_became_peered":"2023-11-24T06:08:58.062992+0000","last_unstale":"2023-11-24T06:08:59.053202+0000","last_undegraded":"2023-11-24T06:08:59.053202+0000","last_fullsized":"2023-11-24T06:08:59.053202+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:12:24.917907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047458+0000","last_change":"2023-11-24T06:08:59.066579+0000","last_active":"2023-11-24T06:09:00.047458+0000","last_peered":"2023-11-24T06:09:00.047458+0000","last_clean":"2023-11-24T06:09:00.047458+0000","last_became_active":"2023-11-24T06:08:59.066234+0000","last_became_peered":"2023-11-24T06:08:59.066234+0000","last_unstale":"2023-11-24T06:09:00.047458+0000","last_undegraded":"2023-11-24T06:09:00.047458+0000","last_fullsized":"2023-11-24T06:09:00.047458+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:37:54.962724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.057071+0000","last_change":"2023-11-24T06:08:58.063623+0000","last_active":"2023-11-24T06:08:59.057071+0000","last_peered":"2023-11-24T06:08:59.057071+0000","last_clean":"2023-11-24T06:08:59.057071+0000","last_became_active":"2023-11-24T06:08:58.063360+0000","last_became_peered":"2023-11-24T06:08:58.063360+0000","last_unstale":"2023-11-24T06:08:59.057071+0000","last_undegraded":"2023-11-24T06:08:59.057071+0000","last_fullsized":"2023-11-24T06:08:59.057071+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:14:13.475472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047199+0000","last_change":"2023-11-24T06:08:59.055431+0000","last_active":"2023-11-24T06:09:00.047199+0000","last_peered":"2023-11-24T06:09:00.047199+0000","last_clean":"2023-11-24T06:09:00.047199+0000","last_became_active":"2023-11-24T06:08:59.054742+0000","last_became_peered":"2023-11-24T06:08:59.054742+0000","last_unstale":"2023-11-24T06:09:00.047199+0000","last_undegraded":"2023-11-24T06:09:00.047199+0000","last_fullsized":"2023-11-24T06:09:00.047199+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:54:30.696914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.081508+0000","last_change":"2023-11-24T06:08:58.064649+0000","last_active":"2023-11-24T06:08:59.081508+0000","last_peered":"2023-11-24T06:08:59.081508+0000","last_clean":"2023-11-24T06:08:59.081508+0000","last_became_active":"2023-11-24T06:08:58.063467+0000","last_became_peered":"2023-11-24T06:08:58.063467+0000","last_unstale":"2023-11-24T06:08:59.081508+0000","last_undegraded":"2023-11-24T06:08:59.081508+0000","last_fullsized":"2023-11-24T06:08:59.081508+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:10:41.185077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.395320+0000","last_change":"2023-11-24T06:08:59.069413+0000","last_active":"2023-11-24T06:09:00.395320+0000","last_peered":"2023-11-24T06:09:00.395320+0000","last_clean":"2023-11-24T06:09:00.395320+0000","last_became_active":"2023-11-24T06:08:59.069159+0000","last_became_peered":"2023-11-24T06:08:59.069159+0000","last_unstale":"2023-11-24T06:09:00.395320+0000","last_undegraded":"2023-11-24T06:09:00.395320+0000","last_fullsized":"2023-11-24T06:09:00.395320+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:57.493230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.075488+0000","last_change":"2023-11-24T06:08:58.064641+0000","last_active":"2023-11-24T06:08:59.075488+0000","last_peered":"2023-11-24T06:08:59.075488+0000","last_clean":"2023-11-24T06:08:59.075488+0000","last_became_active":"2023-11-24T06:08:58.064384+0000","last_became_peered":"2023-11-24T06:08:58.064384+0000","last_unstale":"2023-11-24T06:08:59.075488+0000","last_undegraded":"2023-11-24T06:08:59.075488+0000","last_fullsized":"2023-11-24T06:08:59.075488+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:42:14.623811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.394811+0000","last_change":"2023-11-24T06:08:59.061983+0000","last_active":"2023-11-24T06:09:00.394811+0000","last_peered":"2023-11-24T06:09:00.394811+0000","last_clean":"2023-11-24T06:09:00.394811+0000","last_became_active":"2023-11-24T06:08:59.061777+0000","last_became_peered":"2023-11-24T06:08:59.061777+0000","last_unstale":"2023-11-24T06:09:00.394811+0000","last_undegraded":"2023-11-24T06:09:00.394811+0000","last_fullsized":"2023-11-24T06:09:00.394811+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:46:44.218249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.060028+0000","last_change":"2023-11-24T06:08:58.053451+0000","last_active":"2023-11-24T06:08:59.060028+0000","last_peered":"2023-11-24T06:08:59.060028+0000","last_clean":"2023-11-24T06:08:59.060028+0000","last_became_active":"2023-11-24T06:08:58.053201+0000","last_became_peered":"2023-11-24T06:08:58.053201+0000","last_unstale":"2023-11-24T06:08:59.060028+0000","last_undegraded":"2023-11-24T06:08:59.060028+0000","last_fullsized":"2023-11-24T06:08:59.060028+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:06:04.974056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.129889+0000","last_change":"2023-11-24T06:08:59.061789+0000","last_active":"2023-11-24T06:09:00.129889+0000","last_peered":"2023-11-24T06:09:00.129889+0000","last_clean":"2023-11-24T06:09:00.129889+0000","last_became_active":"2023-11-24T06:08:59.061504+0000","last_became_peered":"2023-11-24T06:08:59.061504+0000","last_unstale":"2023-11-24T06:09:00.129889+0000","last_undegraded":"2023-11-24T06:09:00.129889+0000","last_fullsized":"2023-11-24T06:09:00.129889+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:16:02.211633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.064335+0000","last_change":"2023-11-24T06:08:58.063749+0000","last_active":"2023-11-24T06:08:59.064335+0000","last_peered":"2023-11-24T06:08:59.064335+0000","last_clean":"2023-11-24T06:08:59.064335+0000","last_became_active":"2023-11-24T06:08:58.063432+0000","last_became_peered":"2023-11-24T06:08:58.063432+0000","last_unstale":"2023-11-24T06:08:59.064335+0000","last_undegraded":"2023-11-24T06:08:59.064335+0000","last_fullsized":"2023-11-24T06:08:59.064335+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:42:12.531071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.393812+0000","last_change":"2023-11-24T06:08:59.065911+0000","last_active":"2023-11-24T06:09:00.393812+0000","last_peered":"2023-11-24T06:09:00.393812+0000","last_clean":"2023-11-24T06:09:00.393812+0000","last_became_active":"2023-11-24T06:08:59.065634+0000","last_became_peered":"2023-11-24T06:08:59.065634+0000","last_unstale":"2023-11-24T06:09:00.393812+0000","last_undegraded":"2023-11-24T06:09:00.393812+0000","last_fullsized":"2023-11-24T06:09:00.393812+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:34:40.401608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.053879+0000","last_change":"2023-11-24T06:08:58.054098+0000","last_active":"2023-11-24T06:08:59.053879+0000","last_peered":"2023-11-24T06:08:59.053879+0000","last_clean":"2023-11-24T06:08:59.053879+0000","last_became_active":"2023-11-24T06:08:58.053759+0000","last_became_peered":"2023-11-24T06:08:58.053759+0000","last_unstale":"2023-11-24T06:08:59.053879+0000","last_undegraded":"2023-11-24T06:08:59.053879+0000","last_fullsized":"2023-11-24T06:08:59.053879+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:12:24.917907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.393760+0000","last_change":"2023-11-24T06:08:59.054606+0000","last_active":"2023-11-24T06:09:00.393760+0000","last_peered":"2023-11-24T06:09:00.393760+0000","last_clean":"2023-11-24T06:09:00.393760+0000","last_became_active":"2023-11-24T06:08:59.054302+0000","last_became_peered":"2023-11-24T06:08:59.054302+0000","last_unstale":"2023-11-24T06:09:00.393760+0000","last_undegraded":"2023-11-24T06:09:00.393760+0000","last_fullsized":"2023-11-24T06:09:00.393760+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:02:20.298388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.070679+0000","last_change":"2023-11-24T06:08:58.064790+0000","last_active":"2023-11-24T06:08:59.070679+0000","last_peered":"2023-11-24T06:08:59.070679+0000","last_clean":"2023-11-24T06:08:59.070679+0000","last_became_active":"2023-11-24T06:08:58.064547+0000","last_became_peered":"2023-11-24T06:08:58.064547+0000","last_unstale":"2023-11-24T06:08:59.070679+0000","last_undegraded":"2023-11-24T06:08:59.070679+0000","last_fullsized":"2023-11-24T06:08:59.070679+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:11:11.810533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.393797+0000","last_change":"2023-11-24T06:08:59.063729+0000","last_active":"2023-11-24T06:09:00.393797+0000","last_peered":"2023-11-24T06:09:00.393797+0000","last_clean":"2023-11-24T06:09:00.393797+0000","last_became_active":"2023-11-24T06:08:59.063456+0000","last_became_peered":"2023-11-24T06:08:59.063456+0000","last_unstale":"2023-11-24T06:09:00.393797+0000","last_undegraded":"2023-11-24T06:09:00.393797+0000","last_fullsized":"2023-11-24T06:09:00.393797+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:34:55.275842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.046361+0000","last_change":"2023-11-24T06:08:58.064866+0000","last_active":"2023-11-24T06:08:59.046361+0000","last_peered":"2023-11-24T06:08:59.046361+0000","last_clean":"2023-11-24T06:08:59.046361+0000","last_became_active":"2023-11-24T06:08:58.064641+0000","last_became_peered":"2023-11-24T06:08:58.064641+0000","last_unstale":"2023-11-24T06:08:59.046361+0000","last_undegraded":"2023-11-24T06:08:59.046361+0000","last_fullsized":"2023-11-24T06:08:59.046361+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:55:31.418034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.393761+0000","last_change":"2023-11-24T06:08:59.063783+0000","last_active":"2023-11-24T06:09:00.393761+0000","last_peered":"2023-11-24T06:09:00.393761+0000","last_clean":"2023-11-24T06:09:00.393761+0000","last_became_active":"2023-11-24T06:08:59.063490+0000","last_became_peered":"2023-11-24T06:08:59.063490+0000","last_unstale":"2023-11-24T06:09:00.393761+0000","last_undegraded":"2023-11-24T06:09:00.393761+0000","last_fullsized":"2023-11-24T06:09:00.393761+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:50:42.145844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.051083+0000","last_change":"2023-11-24T06:08:58.063865+0000","last_active":"2023-11-24T06:08:59.051083+0000","last_peered":"2023-11-24T06:08:59.051083+0000","last_clean":"2023-11-24T06:08:59.051083+0000","last_became_active":"2023-11-24T06:08:58.063559+0000","last_became_peered":"2023-11-24T06:08:58.063559+0000","last_unstale":"2023-11-24T06:08:59.051083+0000","last_undegraded":"2023-11-24T06:08:59.051083+0000","last_fullsized":"2023-11-24T06:08:59.051083+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:37:09.304240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047482+0000","last_change":"2023-11-24T06:08:59.063018+0000","last_active":"2023-11-24T06:09:00.047482+0000","last_peered":"2023-11-24T06:09:00.047482+0000","last_clean":"2023-11-24T06:09:00.047482+0000","last_became_active":"2023-11-24T06:08:59.062589+0000","last_became_peered":"2023-11-24T06:08:59.062589+0000","last_unstale":"2023-11-24T06:09:00.047482+0000","last_undegraded":"2023-11-24T06:09:00.047482+0000","last_fullsized":"2023-11-24T06:09:00.047482+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:09:48.969762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.048802+0000","last_change":"2023-11-24T06:08:58.063649+0000","last_active":"2023-11-24T06:08:59.048802+0000","last_peered":"2023-11-24T06:08:59.048802+0000","last_clean":"2023-11-24T06:08:59.048802+0000","last_became_active":"2023-11-24T06:08:58.063374+0000","last_became_peered":"2023-11-24T06:08:58.063374+0000","last_unstale":"2023-11-24T06:08:59.048802+0000","last_undegraded":"2023-11-24T06:08:59.048802+0000","last_fullsized":"2023-11-24T06:08:59.048802+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:37:09.304240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.393842+0000","last_change":"2023-11-24T06:08:59.060563+0000","last_active":"2023-11-24T06:09:00.393842+0000","last_peered":"2023-11-24T06:09:00.393842+0000","last_clean":"2023-11-24T06:09:00.393842+0000","last_became_active":"2023-11-24T06:08:59.060227+0000","last_became_peered":"2023-11-24T06:08:59.060227+0000","last_unstale":"2023-11-24T06:09:00.393842+0000","last_undegraded":"2023-11-24T06:09:00.393842+0000","last_fullsized":"2023-11-24T06:09:00.393842+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:14:11.383611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.069434+0000","last_change":"2023-11-24T06:08:58.063031+0000","last_active":"2023-11-24T06:08:59.069434+0000","last_peered":"2023-11-24T06:08:59.069434+0000","last_clean":"2023-11-24T06:08:59.069434+0000","last_became_active":"2023-11-24T06:08:58.062737+0000","last_became_peered":"2023-11-24T06:08:58.062737+0000","last_unstale":"2023-11-24T06:08:59.069434+0000","last_undegraded":"2023-11-24T06:08:59.069434+0000","last_fullsized":"2023-11-24T06:08:59.069434+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T16:42:12.531071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.394708+0000","last_change":"2023-11-24T06:08:59.075782+0000","last_active":"2023-11-24T06:09:00.394708+0000","last_peered":"2023-11-24T06:09:00.394708+0000","last_clean":"2023-11-24T06:09:00.394708+0000","last_became_active":"2023-11-24T06:08:59.075494+0000","last_became_peered":"2023-11-24T06:08:59.075494+0000","last_unstale":"2023-11-24T06:09:00.394708+0000","last_undegraded":"2023-11-24T06:09:00.394708+0000","last_fullsized":"2023-11-24T06:09:00.394708+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:09:48.969762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.127999+0000","last_change":"2023-11-24T06:08:58.065112+0000","last_active":"2023-11-24T06:09:00.127999+0000","last_peered":"2023-11-24T06:09:00.127999+0000","last_clean":"2023-11-24T06:09:00.127999+0000","last_became_active":"2023-11-24T06:08:58.064831+0000","last_became_peered":"2023-11-24T06:08:58.064831+0000","last_unstale":"2023-11-24T06:09:00.127999+0000","last_undegraded":"2023-11-24T06:09:00.127999+0000","last_fullsized":"2023-11-24T06:09:00.127999+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:17:31.119037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.127928+0000","last_change":"2023-11-24T06:08:59.056358+0000","last_active":"2023-11-24T06:09:00.127928+0000","last_peered":"2023-11-24T06:09:00.127928+0000","last_clean":"2023-11-24T06:09:00.127928+0000","last_became_active":"2023-11-24T06:08:59.056099+0000","last_became_peered":"2023-11-24T06:08:59.056099+0000","last_unstale":"2023-11-24T06:09:00.127928+0000","last_undegraded":"2023-11-24T06:09:00.127928+0000","last_fullsized":"2023-11-24T06:09:00.127928+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:42:15.629420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.053961+0000","last_change":"2023-11-24T06:08:58.063743+0000","last_active":"2023-11-24T06:08:59.053961+0000","last_peered":"2023-11-24T06:08:59.053961+0000","last_clean":"2023-11-24T06:08:59.053961+0000","last_became_active":"2023-11-24T06:08:58.063471+0000","last_became_peered":"2023-11-24T06:08:58.063471+0000","last_unstale":"2023-11-24T06:08:59.053961+0000","last_undegraded":"2023-11-24T06:08:59.053961+0000","last_fullsized":"2023-11-24T06:08:59.053961+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:14:13.475472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047092+0000","last_change":"2023-11-24T06:08:59.058266+0000","last_active":"2023-11-24T06:09:00.047092+0000","last_peered":"2023-11-24T06:09:00.047092+0000","last_clean":"2023-11-24T06:09:00.047092+0000","last_became_active":"2023-11-24T06:08:59.057972+0000","last_became_peered":"2023-11-24T06:08:59.057972+0000","last_unstale":"2023-11-24T06:09:00.047092+0000","last_undegraded":"2023-11-24T06:09:00.047092+0000","last_fullsized":"2023-11-24T06:09:00.047092+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:11:12.816142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.050086+0000","last_change":"2023-11-24T06:08:58.058135+0000","last_active":"2023-11-24T06:08:59.050086+0000","last_peered":"2023-11-24T06:08:59.050086+0000","last_clean":"2023-11-24T06:08:59.050086+0000","last_became_active":"2023-11-24T06:08:58.057817+0000","last_became_peered":"2023-11-24T06:08:58.057817+0000","last_unstale":"2023-11-24T06:08:59.050086+0000","last_undegraded":"2023-11-24T06:08:59.050086+0000","last_fullsized":"2023-11-24T06:08:59.050086+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T12:55:31.418034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.128046+0000","last_change":"2023-11-24T06:08:59.059598+0000","last_active":"2023-11-24T06:09:00.128046+0000","last_peered":"2023-11-24T06:09:00.128046+0000","last_clean":"2023-11-24T06:09:00.128046+0000","last_became_active":"2023-11-24T06:08:59.059352+0000","last_became_peered":"2023-11-24T06:08:59.059352+0000","last_unstale":"2023-11-24T06:09:00.128046+0000","last_undegraded":"2023-11-24T06:09:00.128046+0000","last_fullsized":"2023-11-24T06:09:00.128046+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T08:12:25.923516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.053071+0000","last_change":"2023-11-24T06:08:58.064104+0000","last_active":"2023-11-24T06:08:59.053071+0000","last_peered":"2023-11-24T06:08:59.053071+0000","last_clean":"2023-11-24T06:08:59.053071+0000","last_became_active":"2023-11-24T06:08:58.062769+0000","last_became_peered":"2023-11-24T06:08:58.062769+0000","last_unstale":"2023-11-24T06:08:59.053071+0000","last_undegraded":"2023-11-24T06:08:59.053071+0000","last_fullsized":"2023-11-24T06:08:59.053071+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:06:04.974056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.048241+0000","last_change":"2023-11-24T06:08:59.076400+0000","last_active":"2023-11-24T06:09:00.048241+0000","last_peered":"2023-11-24T06:09:00.048241+0000","last_clean":"2023-11-24T06:09:00.048241+0000","last_became_active":"2023-11-24T06:08:59.076118+0000","last_became_peered":"2023-11-24T06:08:59.076118+0000","last_unstale":"2023-11-24T06:09:00.048241+0000","last_undegraded":"2023-11-24T06:09:00.048241+0000","last_fullsized":"2023-11-24T06:09:00.048241+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T13:46:44.218249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.051240+0000","last_change":"2023-11-24T06:08:58.057018+0000","last_active":"2023-11-24T06:08:59.051240+0000","last_peered":"2023-11-24T06:08:59.051240+0000","last_clean":"2023-11-24T06:08:59.051240+0000","last_became_active":"2023-11-24T06:08:58.056773+0000","last_became_peered":"2023-11-24T06:08:58.056773+0000","last_unstale":"2023-11-24T06:08:59.051240+0000","last_undegraded":"2023-11-24T06:08:59.051240+0000","last_fullsized":"2023-11-24T06:08:59.051240+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:14:10.378002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.394670+0000","last_change":"2023-11-24T06:08:59.066548+0000","last_active":"2023-11-24T06:09:00.394670+0000","last_peered":"2023-11-24T06:09:00.394670+0000","last_clean":"2023-11-24T06:09:00.394670+0000","last_became_active":"2023-11-24T06:08:59.066262+0000","last_became_peered":"2023-11-24T06:08:59.066262+0000","last_unstale":"2023-11-24T06:09:00.394670+0000","last_undegraded":"2023-11-24T06:09:00.394670+0000","last_fullsized":"2023-11-24T06:09:00.394670+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T10:56:24.331211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.060301+0000","last_change":"2023-11-24T06:08:58.061473+0000","last_active":"2023-11-24T06:08:59.060301+0000","last_peered":"2023-11-24T06:08:59.060301+0000","last_clean":"2023-11-24T06:08:59.060301+0000","last_became_active":"2023-11-24T06:08:58.061183+0000","last_became_peered":"2023-11-24T06:08:58.061183+0000","last_unstale":"2023-11-24T06:08:59.060301+0000","last_undegraded":"2023-11-24T06:08:59.060301+0000","last_fullsized":"2023-11-24T06:08:59.060301+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T07:09:47.964153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047573+0000","last_change":"2023-11-24T06:08:59.065850+0000","last_active":"2023-11-24T06:09:00.047573+0000","last_peered":"2023-11-24T06:09:00.047573+0000","last_clean":"2023-11-24T06:09:00.047573+0000","last_became_active":"2023-11-24T06:08:59.065580+0000","last_became_peered":"2023-11-24T06:08:59.065580+0000","last_unstale":"2023-11-24T06:09:00.047573+0000","last_undegraded":"2023-11-24T06:09:00.047573+0000","last_fullsized":"2023-11-24T06:09:00.047573+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:58.034363+0000","last_clean_scrub_stamp":"2023-11-24T06:08:58.034363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T09:37:57.493230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.046251+0000","last_change":"2023-11-24T06:08:58.054098+0000","last_active":"2023-11-24T06:08:59.046251+0000","last_peered":"2023-11-24T06:08:59.046251+0000","last_clean":"2023-11-24T06:08:59.046251+0000","last_became_active":"2023-11-24T06:08:58.053776+0000","last_became_peered":"2023-11-24T06:08:58.053776+0000","last_unstale":"2023-11-24T06:08:59.046251+0000","last_undegraded":"2023-11-24T06:08:59.046251+0000","last_fullsized":"2023-11-24T06:08:59.046251+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:08:57.028754+0000","last_clean_scrub_stamp":"2023-11-24T06:08:57.028754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T17:14:10.378002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":177,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-11-24T06:08:59.049077+0000","last_change":"2023-11-24T06:07:49.463719+0000","last_active":"2023-11-24T06:08:59.049077+0000","last_peered":"2023-11-24T06:08:59.049077+0000","last_clean":"2023-11-24T06:08:59.049077+0000","last_became_active":"2023-11-24T06:07:49.462736+0000","last_became_peered":"2023-11-24T06:07:49.462736+0000","last_unstale":"2023-11-24T06:08:59.049077+0000","last_undegraded":"2023-11-24T06:08:59.049077+0000","last_fullsized":"2023-11-24T06:08:59.049077+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_clean_scrub_stamp":"2023-11-24T06:07:47.860604+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:16.508770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":983040,"data_stored":742365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.40000000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.53000000000000003}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.441}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.46800000000000003}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.38},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.49299999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.61399999999999999}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.41799999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":393216,"data_stored":152523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60999999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.65700000000000003}]},{"osd":2,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.44600000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.54300000000000004}]},{"osd":4,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.38},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.42099999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.624}]},{"osd":7,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.64400000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":153541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.7670000000000003,"5min":4.7670000000000003,"15min":4.7670000000000003},"last":0.217},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.0810000000000004,"5min":6.0810000000000004,"15min":6.0810000000000004},"last":0.315}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.8340000000000001,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.27900000000000003}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.34699999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.9529999999999998,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.64500000000000002}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.9640000000000004,"5min":5.9640000000000004,"15min":5.9640000000000004},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":3.02,"5min":3.02,"15min":3.02},"last":0.65700000000000003}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":6.048,"5min":6.048,"15min":6.048},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.9929999999999999,"5min":2.9929999999999999,"15min":2.9929999999999999},"last":0.63700000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":6.0129999999999999,"5min":6.0129999999999999,"15min":6.0129999999999999},"last":0.57899999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":978944,"data_stored":742275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":12.076000000000001,"5min":12.076000000000001,"15min":12.076000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":4.181,"5min":4.181,"15min":4.181},"last":0.46700000000000003}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.2089999999999996,"5min":4.2089999999999996,"15min":4.2089999999999996},"last":0.50800000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.56100000000000005}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":4.1520000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.65900000000000003}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":4.1260000000000003,"5min":4.1260000000000003,"15min":4.1260000000000003},"last":0.69899999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.78500000000000003}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.77200000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":151977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.7379999999999995,"5min":9.7379999999999995,"15min":9.7379999999999995},"last":0.32000000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":11.967000000000001,"5min":11.967000000000001,"15min":11.967000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":9.593,"5min":9.593,"15min":9.593},"last":0.38900000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.375},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.36299999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":9.6479999999999997,"5min":9.6479999999999997,"15min":9.6479999999999997},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.60999999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":0.621},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":9.7119999999999997,"5min":9.7119999999999997,"15min":9.7119999999999997},"last":0.74099999999999999}]},{"osd":6,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.76300000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":9.6859999999999999,"5min":9.6859999999999999,"15min":9.6859999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.67000000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":153541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.219},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.29499999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.251}]},{"osd":2,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.32100000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.52400000000000002}]},{"osd":5,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.435},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.53400000000000003}]},{"osd":6,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.496},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.47599999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.46500000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":393216,"data_stored":152011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.9539999999999997,"5min":6.9539999999999997,"15min":6.9539999999999997},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":7.0039999999999996,"5min":7.0039999999999996,"15min":7.0039999999999996},"last":0.745}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":12.907,"5min":12.907,"15min":12.907},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":6.9710000000000001,"5min":6.9710000000000001,"15min":6.9710000000000001},"last":0.76900000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.79500000000000004}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":7.0540000000000003,"5min":7.0540000000000003,"15min":7.0540000000000003},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":7.0679999999999996,"5min":7.0679999999999996,"15min":7.0679999999999996},"last":0.78200000000000003}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.63700000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.69999999999999996}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.72899999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":393216,"data_stored":152579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.64800000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.63800000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.996,"5min":1.996,"15min":1.996},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.76700000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.80300000000000005}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.502},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.53800000000000003}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.58499999999999996}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.67100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:09:03.557 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-11-24T06:09:03.558 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-11-24T06:09:03.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.718+0000 7fe63b73a700 1 Processor -- start 2023-11-24T06:09:03.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.718+0000 7fe63b73a700 1 -- start start 2023-11-24T06:09:03.724 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.719+0000 7fe63b73a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe6340b3ab0 0x7fe6340b4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:03.724 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.719+0000 7fe63b73a700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6340ce5b0 con 0x7fe63415add0 2023-11-24T06:09:03.724 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.719+0000 7fe63b73a700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe634072d70 con 0x7fe63406f540 2023-11-24T06:09:03.724 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.719+0000 7fe63b73a700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6340b4f10 con 0x7fe6340b3ab0 2023-11-24T06:09:03.724 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.719+0000 7fe6394d6700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fe63415add0 0x7fe63406ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54676/0 (socket says 172.21.15.91:54676) 2023-11-24T06:09:03.724 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.719+0000 7fe6394d6700 1 -- 172.21.15.91:0/3643202473 learned_addr learned my addr 172.21.15.91:0/3643202473 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:03.724 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.719+0000 7fe639cd7700 1 --2- 172.21.15.91:0/3643202473 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe6340b3ab0 0x7fe6340b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:03.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.719+0000 7fe62bfff700 1 -- 172.21.15.91:0/3643202473 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419937154 0 0) 0x7fe6340ce5b0 con 0x7fe63415add0 2023-11-24T06:09:03.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.719+0000 7fe62bfff700 1 -- 172.21.15.91:0/3643202473 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6200035b0 con 0x7fe63415add0 2023-11-24T06:09:03.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe62bfff700 1 -- 172.21.15.91:0/3643202473 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1059496126 0 0) 0x7fe6200035b0 con 0x7fe63415add0 2023-11-24T06:09:03.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe62bfff700 1 -- 172.21.15.91:0/3643202473 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe6340b3ab0 msgr2=0x7fe6340b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:03.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe62bfff700 1 --2- 172.21.15.91:0/3643202473 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe6340b3ab0 0x7fe6340b4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe62bfff700 1 -- 172.21.15.91:0/3643202473 >> v1:172.21.15.94:6789/0 conn(0x7fe63406f540 legacy=0x7fe6340cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:03.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe62bfff700 1 -- 172.21.15.91:0/3643202473 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6340b51f0 con 0x7fe63415add0 2023-11-24T06:09:03.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe62bfff700 1 -- 172.21.15.91:0/3643202473 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7fe630001670 con 0x7fe63415add0 2023-11-24T06:09:03.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe62bfff700 1 -- 172.21.15.91:0/3643202473 >> v1:172.21.15.91:6789/0 conn(0x7fe63415add0 legacy=0x7fe63406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:03.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe62bfff700 1 --2- 172.21.15.91:0/3643202473 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe620003a30 0x7fe620003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:03.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe62bfff700 1 --2- 172.21.15.91:0/3643202473 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe6340b3ab0 0x7fe620004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:03.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe62bfff700 1 --2- 172.21.15.91:0/3643202473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe620004d90 0x7fe620005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:03.725 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe62bfff700 1 -- 172.21.15.91:0/3643202473 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6340b4f10 con 0x7fe620003a30 2023-11-24T06:09:03.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe62bfff700 1 -- 172.21.15.91:0/3643202473 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62000dc50 con 0x7fe620004d90 2023-11-24T06:09:03.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe62bfff700 1 -- 172.21.15.91:0/3643202473 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62000de10 con 0x7fe6340b3ab0 2023-11-24T06:09:03.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe6394d6700 1 --2- 172.21.15.91:0/3643202473 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe620003a30 0x7fe620003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:03.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe639cd7700 1 --2- 172.21.15.91:0/3643202473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe620004d90 0x7fe620005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:03.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe638cd5700 1 --2- 172.21.15.91:0/3643202473 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe6340b3ab0 0x7fe620004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:03.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe6394d6700 1 -- 172.21.15.91:0/3643202473 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe6340b3ab0 msgr2=0x7fe620004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:03.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe6394d6700 1 --2- 172.21.15.91:0/3643202473 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe6340b3ab0 0x7fe620004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe6394d6700 1 -- 172.21.15.91:0/3643202473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe620004d90 msgr2=0x7fe620005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:03.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe6394d6700 1 --2- 172.21.15.91:0/3643202473 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe620004d90 0x7fe620005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe6394d6700 1 -- 172.21.15.91:0/3643202473 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6200035b0 con 0x7fe620003a30 2023-11-24T06:09:03.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe6394d6700 1 --2- 172.21.15.91:0/3643202473 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe620003a30 0x7fe620003e10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe63000c2f0 tx=0x7fe620005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4868265839f6ecc server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:03.726 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe62bfff700 1 -- 172.21.15.91:0/3643202473 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63000cd70 con 0x7fe620003a30 2023-11-24T06:09:03.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe62bfff700 1 -- 172.21.15.91:0/3643202473 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe630005960 con 0x7fe620003a30 2023-11-24T06:09:03.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.720+0000 7fe62bfff700 1 -- 172.21.15.91:0/3643202473 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630005cf0 con 0x7fe620003a30 2023-11-24T06:09:03.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.721+0000 7fe63b73a700 1 -- 172.21.15.91:0/3643202473 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe620003a30 msgr2=0x7fe620003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:03.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.721+0000 7fe63b73a700 1 --2- 172.21.15.91:0/3643202473 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe620003a30 0x7fe620003e10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fe63000c2f0 tx=0x7fe620005880 comp rx=0 tx=0).stop 2023-11-24T06:09:03.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.721+0000 7fe639cd7700 1 -- 172.21.15.91:0/3643202473 reap_dead start 2023-11-24T06:09:03.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.721+0000 7fe63b73a700 1 -- 172.21.15.91:0/3643202473 shutdown_connections 2023-11-24T06:09:03.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.721+0000 7fe63b73a700 1 -- 172.21.15.91:0/3643202473 >> 172.21.15.91:0/3643202473 conn(0x7fe6340b27b0 msgr2=0x7fe6340b2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:03.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.721+0000 7fe63b73a700 1 -- 172.21.15.91:0/3643202473 shutdown_connections 2023-11-24T06:09:03.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.721+0000 7fe63b73a700 1 -- 172.21.15.91:0/3643202473 wait complete. 2023-11-24T06:09:03.730 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.723+0000 7fe63b73a700 1 Processor -- start 2023-11-24T06:09:03.730 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.723+0000 7fe63b73a700 1 -- start start 2023-11-24T06:09:03.730 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe63b73a700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe620003a30 0x7fe63414c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:03.730 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe63b73a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe63413c290 0x7fe63413e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:03.730 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe63b73a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe63413c710 0x7fe634156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:03.730 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe63b73a700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63413edc0 con 0x7fe620003a30 2023-11-24T06:09:03.730 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe63b73a700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63413f0a0 con 0x7fe63413c710 2023-11-24T06:09:03.730 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe63b73a700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63414f3c0 con 0x7fe63413c290 2023-11-24T06:09:03.730 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe6394d6700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe620003a30 0x7fe63414c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:03.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe6394d6700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe620003a30 0x7fe63414c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59754/0 (socket says 172.21.15.91:59754) 2023-11-24T06:09:03.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe6394d6700 1 -- 172.21.15.91:0/2407862399 learned_addr learned my addr 172.21.15.91:0/2407862399 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:03.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe638cd5700 1 --2- 172.21.15.91:0/2407862399 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe63413c290 0x7fe63413e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:03.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe639cd7700 1 --2- 172.21.15.91:0/2407862399 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe63413c710 0x7fe634156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:03.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe6394d6700 1 -- 172.21.15.91:0/2407862399 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe63413c290 msgr2=0x7fe63413e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:03.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe6394d6700 1 --2- 172.21.15.91:0/2407862399 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe63413c290 0x7fe63413e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe6394d6700 1 -- 172.21.15.91:0/2407862399 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe63413c710 msgr2=0x7fe634156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:03.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe6394d6700 1 --2- 172.21.15.91:0/2407862399 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe63413c710 0x7fe634156330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:03.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe6394d6700 1 -- 172.21.15.91:0/2407862399 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63414f6a0 con 0x7fe620003a30 2023-11-24T06:09:03.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe6394d6700 1 --2- 172.21.15.91:0/2407862399 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe620003a30 0x7fe63414c080 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe62000dc20 tx=0x7fe630001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:03.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe629ffb700 1 -- 172.21.15.91:0/2407862399 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630004870 con 0x7fe620003a30 2023-11-24T06:09:03.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe629ffb700 1 -- 172.21.15.91:0/2407862399 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6300049d0 con 0x7fe620003a30 2023-11-24T06:09:03.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe629ffb700 1 -- 172.21.15.91:0/2407862399 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630004d80 con 0x7fe620003a30 2023-11-24T06:09:03.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.724+0000 7fe63b73a700 1 -- 172.21.15.91:0/2407862399 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe63413edc0 con 0x7fe620003a30 2023-11-24T06:09:03.732 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.725+0000 7fe63b73a700 1 -- 172.21.15.91:0/2407862399 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe63414f0b0 con 0x7fe620003a30 2023-11-24T06:09:03.732 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.725+0000 7fe63b73a700 1 -- 172.21.15.91:0/2407862399 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe63413f260 con 0x7fe620003a30 2023-11-24T06:09:03.732 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.726+0000 7fe629ffb700 1 -- 172.21.15.91:0/2407862399 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe6300070d0 con 0x7fe620003a30 2023-11-24T06:09:03.732 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.726+0000 7fe629ffb700 1 --2- 172.21.15.91:0/2407862399 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe61806aed0 0x7fe61806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:03.732 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.726+0000 7fe629ffb700 1 -- 172.21.15.91:0/2407862399 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe63008ab20 con 0x7fe620003a30 2023-11-24T06:09:03.732 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.726+0000 7fe638cd5700 1 --2- 172.21.15.91:0/2407862399 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe61806aed0 0x7fe61806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:03.732 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.727+0000 7fe638cd5700 1 --2- 172.21.15.91:0/2407862399 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe61806aed0 0x7fe61806d390 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe624000c00 tx=0x7fe624009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:03.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.731+0000 7fe629ffb700 1 -- 172.21.15.91:0/2407862399 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe630055ca0 con 0x7fe620003a30 2023-11-24T06:09:03.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.996+0000 7fe63b73a700 1 -- 172.21.15.91:0/2407862399 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe634075480 con 0x7fe620003a30 2023-11-24T06:09:04.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:03.998+0000 7fe629ffb700 1 -- 172.21.15.91:0/2407862399 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7fe6300592c0 con 0x7fe620003a30 2023-11-24T06:09:04.001 INFO:teuthology.orchestra.run.smithi091.stdout:[client.0] 2023-11-24T06:09:04.001 INFO:teuthology.orchestra.run.smithi091.stdout: key = AQCIPWBlZntSDRAAdyaD3NYZ7aqLp5+7oaQH+Q== 2023-11-24T06:09:04.001 INFO:teuthology.orchestra.run.smithi091.stdout: caps mds = "allow" 2023-11-24T06:09:04.001 INFO:teuthology.orchestra.run.smithi091.stdout: caps mon = "allow r" 2023-11-24T06:09:04.001 INFO:teuthology.orchestra.run.smithi091.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-11-24T06:09:04.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.000+0000 7fe63b73a700 1 -- 172.21.15.91:0/2407862399 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe61806aed0 msgr2=0x7fe61806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.000+0000 7fe63b73a700 1 --2- 172.21.15.91:0/2407862399 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe61806aed0 0x7fe61806d390 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe624000c00 tx=0x7fe624009040 comp rx=0 tx=0).stop 2023-11-24T06:09:04.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.000+0000 7fe63b73a700 1 -- 172.21.15.91:0/2407862399 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe620003a30 msgr2=0x7fe63414c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.000+0000 7fe63b73a700 1 --2- 172.21.15.91:0/2407862399 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe620003a30 0x7fe63414c080 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fe62000dc20 tx=0x7fe630001a00 comp rx=0 tx=0).stop 2023-11-24T06:09:04.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.000+0000 7fe63b73a700 1 -- 172.21.15.91:0/2407862399 shutdown_connections 2023-11-24T06:09:04.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.000+0000 7fe63b73a700 1 --2- 172.21.15.91:0/2407862399 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe61806aed0 0x7fe61806d390 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.000+0000 7fe63b73a700 1 --2- 172.21.15.91:0/2407862399 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe63413c710 0x7fe634156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.000+0000 7fe63b73a700 1 --2- 172.21.15.91:0/2407862399 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe620003a30 0x7fe63414c080 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.000+0000 7fe63b73a700 1 --2- 172.21.15.91:0/2407862399 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe63413c290 0x7fe63413e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.000+0000 7fe63b73a700 1 -- 172.21.15.91:0/2407862399 >> 172.21.15.91:0/2407862399 conn(0x7fe6340b27b0 msgr2=0x7fe63406edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:04.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.001+0000 7fe63b73a700 1 -- 172.21.15.91:0/2407862399 shutdown_connections 2023-11-24T06:09:04.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.001+0000 7fe63b73a700 1 -- 172.21.15.91:0/2407862399 wait complete. 2023-11-24T06:09:04.004 INFO:teuthology.orchestra.run.smithi091.stderr:updated caps for client.0 2023-11-24T06:09:04.015 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:09:04.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.136+0000 7f9303c7c700 1 Processor -- start 2023-11-24T06:09:04.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.136+0000 7f9303c7c700 1 -- start start 2023-11-24T06:09:04.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9303c7c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f92fc15bde0 0x7f92fc15c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9303c7c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f92fc0cf2b0 0x7f92fc0d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9303c7c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f92fc0ce5c0 0x7f92fc0cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9303c7c700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92fc0cef90 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9303c7c700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92fc0d0a90 con 0x7f92fc0ce5c0 2023-11-24T06:09:04.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9303c7c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc06ee10 con 0x7f92fc15bde0 2023-11-24T06:09:04.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9301217700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f92fc0cf2b0 0x7f92fc0d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9301217700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f92fc0cf2b0 0x7f92fc0d01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59764/0 (socket says 172.21.15.91:59764) 2023-11-24T06:09:04.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9301217700 1 -- 172.21.15.91:0/1580328011 learned_addr learned my addr 172.21.15.91:0/1580328011 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:04.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9301217700 1 -- 172.21.15.91:0/1580328011 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f92fc15bde0 msgr2=0x7f92fc15c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:04.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9301217700 1 --2- 172.21.15.91:0/1580328011 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f92fc15bde0 0x7f92fc15c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9301217700 1 -- 172.21.15.91:0/1580328011 >> v2:172.21.15.94:3300/0 conn(0x7f92fc0ce5c0 msgr2=0x7f92fc0cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:04.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9301217700 1 --2- 172.21.15.91:0/1580328011 >> v2:172.21.15.94:3300/0 conn(0x7f92fc0ce5c0 0x7f92fc0cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9301217700 1 -- 172.21.15.91:0/1580328011 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92fc06f0f0 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9301217700 1 --2- 172.21.15.91:0/1580328011 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f92fc0cf2b0 0x7f92fc0d01b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f92fc15fb60 tx=0x7f92ec009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faf8578f209b918d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:04.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9300a16700 1 -- 172.21.15.91:0/1580328011 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec00b870 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9300a16700 1 -- 172.21.15.91:0/1580328011 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec00b9d0 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.137+0000 7f9300a16700 1 -- 172.21.15.91:0/1580328011 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec00bd60 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.138+0000 7f9303c7c700 1 -- 172.21.15.91:0/1580328011 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f92fc0cf2b0 msgr2=0x7f92fc0d01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.138+0000 7f9303c7c700 1 --2- 172.21.15.91:0/1580328011 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f92fc0cf2b0 0x7f92fc0d01b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f92fc15fb60 tx=0x7f92ec009bd0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.138+0000 7f9303c7c700 1 -- 172.21.15.91:0/1580328011 shutdown_connections 2023-11-24T06:09:04.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.138+0000 7f9303c7c700 1 --2- 172.21.15.91:0/1580328011 >> v2:172.21.15.94:3300/0 conn(0x7f92fc0ce5c0 0x7f92fc0cea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.138+0000 7f9303c7c700 1 --2- 172.21.15.91:0/1580328011 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f92fc15bde0 0x7f92fc15c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.138+0000 7f9303c7c700 1 --2- 172.21.15.91:0/1580328011 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f92fc0cf2b0 0x7f92fc0d01b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.138+0000 7f9303c7c700 1 -- 172.21.15.91:0/1580328011 >> 172.21.15.91:0/1580328011 conn(0x7f92fc0b3720 msgr2=0x7f92fc0b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:04.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.138+0000 7f9303c7c700 1 -- 172.21.15.91:0/1580328011 shutdown_connections 2023-11-24T06:09:04.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.138+0000 7f9303c7c700 1 -- 172.21.15.91:0/1580328011 wait complete. 2023-11-24T06:09:04.147 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.139+0000 7f9303c7c700 1 Processor -- start 2023-11-24T06:09:04.147 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.139+0000 7f9303c7c700 1 -- start start 2023-11-24T06:09:04.147 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.140+0000 7f9303c7c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f92fc0ce5c0 0x7f92fc1598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.147 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.140+0000 7f9303c7c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f92fc0cf2b0 0x7f92fc0b6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.147 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.140+0000 7f9303c7c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92fc15bde0 0x7f92fc0b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.147 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.140+0000 7f9303c7c700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc1f6050 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.147 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.140+0000 7f9303c7c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc0b4940 con 0x7f92fc15bde0 2023-11-24T06:09:04.147 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.140+0000 7f9303c7c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92fc0b4c20 con 0x7f92fc0ce5c0 2023-11-24T06:09:04.148 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.140+0000 7f9301217700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f92fc0cf2b0 0x7f92fc0b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.148 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.140+0000 7f9301217700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f92fc0cf2b0 0x7f92fc0b6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59770/0 (socket says 172.21.15.91:59770) 2023-11-24T06:09:04.148 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.140+0000 7f9301217700 1 -- 172.21.15.91:0/745212272 learned_addr learned my addr 172.21.15.91:0/745212272 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:04.148 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.140+0000 7f9301a18700 1 --2- 172.21.15.91:0/745212272 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f92fc0ce5c0 0x7f92fc1598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.148 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.140+0000 7f9302219700 1 --2- 172.21.15.91:0/745212272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92fc15bde0 0x7f92fc0b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.148 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.140+0000 7f9301217700 1 -- 172.21.15.91:0/745212272 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f92fc0ce5c0 msgr2=0x7f92fc1598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.148 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.140+0000 7f9301217700 1 --2- 172.21.15.91:0/745212272 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f92fc0ce5c0 0x7f92fc1598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.148 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.140+0000 7f9301217700 1 -- 172.21.15.91:0/745212272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92fc15bde0 msgr2=0x7f92fc0b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.148 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.140+0000 7f9301217700 1 --2- 172.21.15.91:0/745212272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92fc15bde0 0x7f92fc0b6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.148 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.140+0000 7f9301217700 1 -- 172.21.15.91:0/745212272 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ec009550 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.148 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.141+0000 7f9301217700 1 --2- 172.21.15.91:0/745212272 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f92fc0cf2b0 0x7f92fc0b6650 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f92ec013000 tx=0x7f92ec004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:04.148 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.141+0000 7f92f27fc700 1 -- 172.21.15.91:0/745212272 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec004310 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.141+0000 7f92f27fc700 1 -- 172.21.15.91:0/745212272 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92ec01f070 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.141+0000 7f9303c7c700 1 -- 172.21.15.91:0/745212272 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92fc0b4f00 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.141+0000 7f92f27fc700 1 -- 172.21.15.91:0/745212272 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92ec0036a0 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.141+0000 7f9303c7c700 1 -- 172.21.15.91:0/745212272 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92fc0b5700 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.142+0000 7f9303c7c700 1 -- 172.21.15.91:0/745212272 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92fc068670 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.142+0000 7f92f27fc700 1 -- 172.21.15.91:0/745212272 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f92ec021030 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.142+0000 7f92f27fc700 1 --2- 172.21.15.91:0/745212272 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f92e006aed0 0x7f92e006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.142+0000 7f92f27fc700 1 -- 172.21.15.91:0/745212272 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92ec08a8d0 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.142+0000 7f9301a18700 1 --2- 172.21.15.91:0/745212272 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f92e006aed0 0x7f92e006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.143+0000 7f9301a18700 1 --2- 172.21.15.91:0/745212272 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f92e006aed0 0x7f92e006d390 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f92f8009970 tx=0x7f92f8009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:04.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.147+0000 7f92f27fc700 1 -- 172.21.15.91:0/745212272 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92ec055a10 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.382 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.379+0000 7f9303c7c700 1 -- 172.21.15.91:0/745212272 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92fc15a780 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.382 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.380+0000 7f92f27fc700 1 -- 172.21.15.91:0/745212272 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4907 (secure 0 0 0) 0x7f92ec015090 con 0x7f92fc0cf2b0 2023-11-24T06:09:04.382 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:04.383 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4706,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1685174039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1685174039},{"type":"v1","addr":"172.21.15.94:6834","nonce":1685174039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4709,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1508344631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1508344631},{"type":"v1","addr":"172.21.15.94:6835","nonce":1508344631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5064,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6836/28114058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6834","nonce":28114058},{"type":"v1","addr":"172.21.15.91:6836","nonce":28114058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:08:58.437951+0000","modified":"2023-11-24T06:09:01.476634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.91:6837/1301930708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":1301930708},{"type":"v1","addr":"172.21.15.91:6837","nonce":1301930708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T06:09:04.385 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.382+0000 7f9303c7c700 1 -- 172.21.15.91:0/745212272 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f92e006aed0 msgr2=0x7f92e006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.385 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.382+0000 7f9303c7c700 1 --2- 172.21.15.91:0/745212272 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f92e006aed0 0x7f92e006d390 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f92f8009970 tx=0x7f92f8009110 comp rx=0 tx=0).stop 2023-11-24T06:09:04.385 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.382+0000 7f9303c7c700 1 -- 172.21.15.91:0/745212272 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f92fc0cf2b0 msgr2=0x7f92fc0b6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.386 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.382+0000 7f9303c7c700 1 --2- 172.21.15.91:0/745212272 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f92fc0cf2b0 0x7f92fc0b6650 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f92ec013000 tx=0x7f92ec004750 comp rx=0 tx=0).stop 2023-11-24T06:09:04.386 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.382+0000 7f9303c7c700 1 -- 172.21.15.91:0/745212272 shutdown_connections 2023-11-24T06:09:04.386 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.382+0000 7f9303c7c700 1 --2- 172.21.15.91:0/745212272 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f92e006aed0 0x7f92e006d390 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.386 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.382+0000 7f9303c7c700 1 --2- 172.21.15.91:0/745212272 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f92fc15bde0 0x7f92fc0b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.386 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.382+0000 7f9303c7c700 1 --2- 172.21.15.91:0/745212272 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f92fc0ce5c0 0x7f92fc1598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.386 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.382+0000 7f9303c7c700 1 --2- 172.21.15.91:0/745212272 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f92fc0cf2b0 0x7f92fc0b6650 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.386 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.382+0000 7f9303c7c700 1 -- 172.21.15.91:0/745212272 >> 172.21.15.91:0/745212272 conn(0x7f92fc0b3720 msgr2=0x7f92fc06fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:04.386 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.382+0000 7f9303c7c700 1 -- 172.21.15.91:0/745212272 shutdown_connections 2023-11-24T06:09:04.386 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.382+0000 7f9303c7c700 1 -- 172.21.15.91:0/745212272 wait complete. 2023-11-24T06:09:04.386 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 16 2023-11-24T06:09:04.799 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-11-24T06:08:58.437951+0000', 'modified': '2023-11-24T06:09:01.476634+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5067}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5067': {'gid': 5067, 'name': 'a', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.91:6837/1301930708', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.91:6835', 'nonce': 1301930708}, {'type': 'v1', 'addr': '172.21.15.91:6837', 'nonce': 1301930708}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-11-24T06:09:04.799 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-11-24T06:09:04.799 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-11-24T06:09:04.923 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.920+0000 7f014e2e6700 1 Processor -- start 2023-11-24T06:09:04.923 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.920+0000 7f014e2e6700 1 -- start start 2023-11-24T06:09:04.925 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f014e2e6700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f01480722c0 0x7f01480b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.925 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f014e2e6700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f01481440f0 0x7f014806e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.925 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f014e2e6700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0148143ec0 con 0x7f0148143ae0 2023-11-24T06:09:04.925 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f014e2e6700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01480b40a0 con 0x7f01481440f0 2023-11-24T06:09:04.925 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f014e2e6700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01480b3ae0 con 0x7f01480722c0 2023-11-24T06:09:04.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f014c883700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f01481440f0 0x7f014806e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f0147fff700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f0148143ae0 0x7f0148071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54710/0 (socket says 172.21.15.91:54710) 2023-11-24T06:09:04.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f0147fff700 1 -- 172.21.15.91:0/3476407434 learned_addr learned my addr 172.21.15.91:0/3476407434 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:04.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f01477fe700 1 --2- 172.21.15.91:0/3476407434 >> v2:172.21.15.94:3300/0 conn(0x7f01480722c0 0x7f01480b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f014c883700 1 -- 172.21.15.91:0/3476407434 >> v2:172.21.15.94:3300/0 conn(0x7f01480722c0 msgr2=0x7f01480b51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:04.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f014c883700 1 --2- 172.21.15.91:0/3476407434 >> v2:172.21.15.94:3300/0 conn(0x7f01480722c0 0x7f01480b51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f014c883700 1 -- 172.21.15.91:0/3476407434 >> v1:172.21.15.91:6789/0 conn(0x7f0148143ae0 legacy=0x7f0148071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f0146ffd700 1 -- 172.21.15.91:0/3476407434 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430727830 0 0) 0x7f0148143ec0 con 0x7f0148143ae0 2023-11-24T06:09:04.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f014c883700 1 -- 172.21.15.91:0/3476407434 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01480b3dc0 con 0x7f01481440f0 2023-11-24T06:09:04.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f01477fe700 1 --2- 172.21.15.91:0/3476407434 >> v2:172.21.15.94:3300/0 conn(0x7f01480722c0 0x7f01480b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:04.926 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f014c883700 1 --2- 172.21.15.91:0/3476407434 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f01481440f0 0x7f014806e510 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f01480d1b00 tx=0x7f014000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537146f4960efc5d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:04.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f0146ffd700 1 -- 172.21.15.91:0/3476407434 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014000e070 con 0x7f01481440f0 2023-11-24T06:09:04.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.921+0000 7f0146ffd700 1 -- 172.21.15.91:0/3476407434 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014000bbd0 con 0x7f01481440f0 2023-11-24T06:09:04.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.922+0000 7f0146ffd700 1 -- 172.21.15.91:0/3476407434 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0140012430 con 0x7f01481440f0 2023-11-24T06:09:04.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.922+0000 7f014e2e6700 1 -- 172.21.15.91:0/3476407434 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f01481440f0 msgr2=0x7f014806e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.922+0000 7f014e2e6700 1 --2- 172.21.15.91:0/3476407434 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f01481440f0 0x7f014806e510 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f01480d1b00 tx=0x7f014000b330 comp rx=0 tx=0).stop 2023-11-24T06:09:04.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.922+0000 7f014e2e6700 1 -- 172.21.15.91:0/3476407434 shutdown_connections 2023-11-24T06:09:04.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.922+0000 7f014e2e6700 1 --2- 172.21.15.91:0/3476407434 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f01481440f0 0x7f014806e510 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.922+0000 7f014e2e6700 1 --2- 172.21.15.91:0/3476407434 >> v2:172.21.15.94:3300/0 conn(0x7f01480722c0 0x7f01480b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.922+0000 7f014e2e6700 1 -- 172.21.15.91:0/3476407434 >> 172.21.15.91:0/3476407434 conn(0x7f01480b2740 msgr2=0x7f01480b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:04.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.922+0000 7f014e2e6700 1 -- 172.21.15.91:0/3476407434 shutdown_connections 2023-11-24T06:09:04.927 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.922+0000 7f014e2e6700 1 -- 172.21.15.91:0/3476407434 wait complete. 2023-11-24T06:09:04.930 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.923+0000 7f014e2e6700 1 Processor -- start 2023-11-24T06:09:04.930 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.923+0000 7f014e2e6700 1 -- start start 2023-11-24T06:09:04.930 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.924+0000 7f014e2e6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01480722c0 0x7f01481395f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.930 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.924+0000 7f014e2e6700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0148143ae0 0x7f01480cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.930 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.924+0000 7f014e2e6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01481440f0 0x7f01480cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.930 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.924+0000 7f014e2e6700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01481f3aa0 con 0x7f0148143ae0 2023-11-24T06:09:04.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.924+0000 7f014e2e6700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01480cb5b0 con 0x7f01480722c0 2023-11-24T06:09:04.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.924+0000 7f014e2e6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014813f8e0 con 0x7f01481440f0 2023-11-24T06:09:04.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.924+0000 7f01477fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0148143ae0 0x7f01480cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.924+0000 7f01477fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0148143ae0 0x7f01480cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59784/0 (socket says 172.21.15.91:59784) 2023-11-24T06:09:04.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.924+0000 7f01477fe700 1 -- 172.21.15.91:0/2537283817 learned_addr learned my addr 172.21.15.91:0/2537283817 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:04.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.924+0000 7f014c883700 1 --2- 172.21.15.91:0/2537283817 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01481440f0 0x7f01480cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.924+0000 7f0147fff700 1 --2- 172.21.15.91:0/2537283817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01480722c0 0x7f01481395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.924+0000 7f01477fe700 1 -- 172.21.15.91:0/2537283817 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01481440f0 msgr2=0x7f01480cf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.924+0000 7f01477fe700 1 --2- 172.21.15.91:0/2537283817 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01481440f0 0x7f01480cf6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.924+0000 7f01477fe700 1 -- 172.21.15.91:0/2537283817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01480722c0 msgr2=0x7f01481395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.925+0000 7f01477fe700 1 --2- 172.21.15.91:0/2537283817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01480722c0 0x7f01481395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.931 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.925+0000 7f01477fe700 1 -- 172.21.15.91:0/2537283817 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0140009a70 con 0x7f0148143ae0 2023-11-24T06:09:04.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.925+0000 7f01477fe700 1 --2- 172.21.15.91:0/2537283817 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0148143ae0 0x7f01480cba40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f013800c750 tx=0x7f013800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:04.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.925+0000 7f0144ff9700 1 -- 172.21.15.91:0/2537283817 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0138011800 con 0x7f0148143ae0 2023-11-24T06:09:04.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.925+0000 7f0144ff9700 1 -- 172.21.15.91:0/2537283817 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0138011960 con 0x7f0148143ae0 2023-11-24T06:09:04.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.925+0000 7f014e2e6700 1 -- 172.21.15.91:0/2537283817 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01480ca9a0 con 0x7f0148143ae0 2023-11-24T06:09:04.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.925+0000 7f0144ff9700 1 -- 172.21.15.91:0/2537283817 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0138011cd0 con 0x7f0148143ae0 2023-11-24T06:09:04.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.925+0000 7f014e2e6700 1 -- 172.21.15.91:0/2537283817 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01480caf60 con 0x7f0148143ae0 2023-11-24T06:09:04.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.926+0000 7f0144ff9700 1 -- 172.21.15.91:0/2537283817 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0138021350 con 0x7f0148143ae0 2023-11-24T06:09:04.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.927+0000 7f0144ff9700 1 --2- 172.21.15.91:0/2537283817 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f013406aed0 0x7f013406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.927+0000 7f0147fff700 1 --2- 172.21.15.91:0/2537283817 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f013406aed0 0x7f013406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.927+0000 7f0144ff9700 1 -- 172.21.15.91:0/2537283817 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01380896a0 con 0x7f0148143ae0 2023-11-24T06:09:04.932 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.927+0000 7f0147fff700 1 --2- 172.21.15.91:0/2537283817 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f013406aed0 0x7f013406d390 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f013c000f60 tx=0x7f013c007be0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:04.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.929+0000 7f015a4dfb80 1 Processor -- start 2023-11-24T06:09:04.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.929+0000 7f015a4dfb80 1 -- start start 2023-11-24T06:09:04.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.929+0000 7f015a4dfb80 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e57296b080 0x55e5729b1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.929+0000 7f015a4dfb80 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572980ed0 0x55e5729b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.929+0000 7f015a4dfb80 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e572968490 0x55e572994e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.929+0000 7f015a4dfb80 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5728de770 con 0x55e572980ed0 2023-11-24T06:09:04.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.929+0000 7f015a4dfb80 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5728c5750 con 0x55e572968490 2023-11-24T06:09:04.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.929+0000 7f015a4dfb80 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5728c3070 con 0x55e57296b080 2023-11-24T06:09:04.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.929+0000 7f01477fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572980ed0 0x55e5729b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.930+0000 7f01477fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572980ed0 0x55e5729b1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59792/0 (socket says 172.21.15.91:59792) 2023-11-24T06:09:04.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.930+0000 7f01477fe700 1 -- 172.21.15.91:0/3759005641 learned_addr learned my addr 172.21.15.91:0/3759005641 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:04.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.930+0000 7f0147fff700 1 --2- 172.21.15.91:0/3759005641 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e57296b080 0x55e5729b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.930+0000 7f01477fe700 1 -- 172.21.15.91:0/3759005641 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e57296b080 msgr2=0x55e5729b1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:04.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.930+0000 7f01477fe700 1 --2- 172.21.15.91:0/3759005641 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e57296b080 0x55e5729b1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.930+0000 7f014c883700 1 --2- 172.21.15.91:0/3759005641 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e572968490 0x55e572994e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.930+0000 7f01477fe700 1 -- 172.21.15.91:0/3759005641 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e572968490 msgr2=0x55e572994e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:04.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.930+0000 7f01477fe700 1 --2- 172.21.15.91:0/3759005641 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e572968490 0x55e572994e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.930+0000 7f01477fe700 1 -- 172.21.15.91:0/3759005641 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e57273b5e0 con 0x55e572980ed0 2023-11-24T06:09:04.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.930+0000 7f01477fe700 1 --2- 172.21.15.91:0/3759005641 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572980ed0 0x55e5729b1d80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0138057fe0 tx=0x7f0138058a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5798d24b85caccc6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:04.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.930+0000 7f012dffb700 1 -- 172.21.15.91:0/3759005641 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013805c430 con 0x55e572980ed0 2023-11-24T06:09:04.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.930+0000 7f012dffb700 1 -- 172.21.15.91:0/3759005641 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0138060670 con 0x55e572980ed0 2023-11-24T06:09:04.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.930+0000 7f012dffb700 1 -- 172.21.15.91:0/3759005641 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0138056680 con 0x55e572980ed0 2023-11-24T06:09:04.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.931+0000 7f015a4dfb80 1 -- 172.21.15.91:0/3759005641 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572980ed0 msgr2=0x55e5729b1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.931+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/3759005641 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572980ed0 0x55e5729b1d80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0138057fe0 tx=0x7f0138058a60 comp rx=0 tx=0).stop 2023-11-24T06:09:04.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.931+0000 7f015a4dfb80 1 -- 172.21.15.91:0/3759005641 shutdown_connections 2023-11-24T06:09:04.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.931+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/3759005641 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e572968490 0x55e572994e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.931+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/3759005641 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e57296b080 0x55e5729b1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.931+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/3759005641 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572980ed0 0x55e5729b1d80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.931+0000 7f015a4dfb80 1 -- 172.21.15.91:0/3759005641 >> 172.21.15.91:0/3759005641 conn(0x55e572899310 msgr2=0x55e5729afe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:04.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.931+0000 7f015a4dfb80 1 -- 172.21.15.91:0/3759005641 shutdown_connections 2023-11-24T06:09:04.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.931+0000 7f015a4dfb80 1 -- 172.21.15.91:0/3759005641 wait complete. 2023-11-24T06:09:04.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.931+0000 7f015a4dfb80 1 -- start start 2023-11-24T06:09:04.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.932+0000 7f015a4dfb80 1 Processor -- start 2023-11-24T06:09:04.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f015a4dfb80 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e572968490 0x55e5729ac920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f015a4dfb80 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e57296b080 0x55e57295f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f015a4dfb80 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572980ed0 0x55e5729a9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f015a4dfb80 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5728be960 con 0x55e572980ed0 2023-11-24T06:09:04.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f015a4dfb80 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e572878510 con 0x55e57296b080 2023-11-24T06:09:04.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f015a4dfb80 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e5728781b0 con 0x55e572968490 2023-11-24T06:09:04.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f015a4dfb80 10 client.0 _ll_register_callbacks cb 0x55e5727a3a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T06:09:04.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f014c883700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572980ed0 0x55e5729a9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f014c883700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572980ed0 0x55e5729a9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59800/0 (socket says 172.21.15.91:59800) 2023-11-24T06:09:04.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f014c883700 1 -- 172.21.15.91:0/4128060342 learned_addr learned my addr 172.21.15.91:0/4128060342 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:04.942 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f01477fe700 1 --2- 172.21.15.91:0/4128060342 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e57296b080 0x55e57295f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.943 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f0147fff700 1 --2- 172.21.15.91:0/4128060342 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e572968490 0x55e5729ac920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.943 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f014c883700 1 -- 172.21.15.91:0/4128060342 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e572968490 msgr2=0x55e5729ac920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:04.943 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f014c883700 1 --2- 172.21.15.91:0/4128060342 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e572968490 0x55e5729ac920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.943 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f014c883700 1 -- 172.21.15.91:0/4128060342 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e57296b080 msgr2=0x55e57295f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:04.943 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f014c883700 1 --2- 172.21.15.91:0/4128060342 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e57296b080 0x55e57295f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.943 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f014c883700 1 -- 172.21.15.91:0/4128060342 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01480caf60 con 0x55e572980ed0 2023-11-24T06:09:04.943 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f0126ffd700 10 client.0 ms_handle_connect on v2:172.21.15.91:3300/0 2023-11-24T06:09:04.943 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f014c883700 1 --2- 172.21.15.91:0/4128060342 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572980ed0 0x55e5729a9150 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f01480d1bd0 tx=0x7f0140004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c808fde6ec89f773 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:04.943 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.933+0000 7f0126ffd700 1 -- 172.21.15.91:0/4128060342 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0140020070 con 0x55e572980ed0 2023-11-24T06:09:04.943 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.934+0000 7f0126ffd700 1 -- 172.21.15.91:0/4128060342 <== mon.0 v2:172.21.15.91:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0140003e20 con 0x55e572980ed0 2023-11-24T06:09:04.943 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.934+0000 7f015a4dfb80 1 -- 172.21.15.91:0/4128060342 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e57295da40 con 0x55e572980ed0 2023-11-24T06:09:04.943 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.934+0000 7f0126ffd700 1 -- 172.21.15.91:0/4128060342 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0140012430 con 0x55e572980ed0 2023-11-24T06:09:04.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.934+0000 7f015a4dfb80 10 client.5250 fetch_fsmap learned FSMap version 16 2023-11-24T06:09:04.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.934+0000 7f015a4dfb80 1 -- 172.21.15.91:0/4128060342 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55e5726e7d90 con 0x55e572980ed0 2023-11-24T06:09:04.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.934+0000 7f0126ffd700 1 -- 172.21.15.91:0/4128060342 <== mon.0 v2:172.21.15.91:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e57295da40 con 0x55e572980ed0 2023-11-24T06:09:04.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.934+0000 7f0126ffd700 1 -- 172.21.15.91:0/4128060342 <== mon.0 v2:172.21.15.91:3300/0 5 ==== fsmap(e 16) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0140012890 con 0x55e572980ed0 2023-11-24T06:09:04.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.934+0000 7f015a4dfb80 10 client.5250 fetch_fsmap finished waiting for FSMap version 16 2023-11-24T06:09:04.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.934+0000 7f015a4dfb80 10 client.5250 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-24T06:09:04.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.934+0000 7f015a4dfb80 20 client.5250 populate_metadata read hostname 'smithi091' 2023-11-24T06:09:04.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.934+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/4128060342 >> [v2:172.21.15.91:6835/1301930708,v1:172.21.15.91:6837/1301930708] conn(0x55e572998bb0 0x55e5729c2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.934+0000 7f015a4dfb80 4 client.5250 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-11-24T06:09:04.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.934+0000 7f015a4dfb80 1 -- 172.21.15.91:0/4128060342 --> [v2:172.21.15.91:6835/1301930708,v1:172.21.15.91:6837/1301930708] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e572939840 con 0x55e572998bb0 2023-11-24T06:09:04.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.934+0000 7f0147fff700 1 --2- 172.21.15.91:0/4128060342 >> [v2:172.21.15.91:6835/1301930708,v1:172.21.15.91:6837/1301930708] conn(0x55e572998bb0 0x55e5729c2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.934+0000 7f0147fff700 1 --2- 172.21.15.91:0/4128060342 >> [v2:172.21.15.91:6835/1301930708,v1:172.21.15.91:6837/1301930708] conn(0x55e572998bb0 0x55e5729c2ee0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=284bd607f472a2f2 server_cookie=1ec36ab86419a40a in_seq=0 out_seq=0 2023-11-24T06:09:04.944 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.934+0000 7f0126ffd700 10 client.5250 ms_handle_connect on v2:172.21.15.91:6835/1301930708 2023-11-24T06:09:04.945 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.935+0000 7f0126ffd700 1 -- 172.21.15.91:0/4128060342 <== mds.0 v2:172.21.15.91:6835/1301930708 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e572939840 con 0x55e572998bb0 2023-11-24T06:09:04.945 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.935+0000 7f0126ffd700 10 client.5250 handle_command_reply: tid=0 2023-11-24T06:09:04.945 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.935+0000 7f015a4dfb80 1 client.5250 shutdown 2023-11-24T06:09:04.945 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.935+0000 7f015a4dfb80 1 -- 172.21.15.91:0/4128060342 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572980ed0 msgr2=0x55e5729a9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.945 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.935+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/4128060342 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572980ed0 0x55e5729a9150 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f01480d1bd0 tx=0x7f0140004b40 comp rx=0 tx=0).stop 2023-11-24T06:09:04.945 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.935+0000 7f015a4dfb80 1 -- 172.21.15.91:0/4128060342 shutdown_connections 2023-11-24T06:09:04.945 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.935+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/4128060342 >> [v2:172.21.15.91:6835/1301930708,v1:172.21.15.91:6837/1301930708] conn(0x55e572998bb0 0x55e5729c2ee0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.945 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.935+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/4128060342 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572980ed0 0x55e5729a9150 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.945 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.935+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/4128060342 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e572968490 0x55e5729ac920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.945 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.935+0000 7f0126ffd700 0 client.5250 ms_handle_reset on v2:172.21.15.91:6835/1301930708 2023-11-24T06:09:04.945 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.935+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/4128060342 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e57296b080 0x55e57295f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.945 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.935+0000 7f015a4dfb80 1 -- 172.21.15.91:0/4128060342 >> 172.21.15.91:0/4128060342 conn(0x55e572899310 msgr2=0x55e5729b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:04.945 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.935+0000 7f015a4dfb80 1 -- 172.21.15.91:0/4128060342 shutdown_connections 2023-11-24T06:09:04.946 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.935+0000 7f015a4dfb80 1 -- 172.21.15.91:0/4128060342 wait complete. 2023-11-24T06:09:04.946 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.935+0000 7f015a4dfb80 20 client.5250 trim_cache size 0 max 16384 2023-11-24T06:09:04.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.953+0000 7f015a4dfb80 1 Processor -- start 2023-11-24T06:09:04.956 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.953+0000 7f015a4dfb80 1 -- start start 2023-11-24T06:09:04.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.954+0000 7f015a4dfb80 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572968490 0x55e572999050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.954+0000 7f015a4dfb80 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e57296b080 0x55e5729982e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.954+0000 7f015a4dfb80 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e572980ed0 0x55e57297ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.954+0000 7f015a4dfb80 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5729602f0 con 0x55e572968490 2023-11-24T06:09:04.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.954+0000 7f015a4dfb80 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e57273d8d0 con 0x55e57296b080 2023-11-24T06:09:04.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.954+0000 7f015a4dfb80 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e57296ac50 con 0x55e572980ed0 2023-11-24T06:09:04.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.954+0000 7f0147fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572968490 0x55e572999050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.954+0000 7f0147fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572968490 0x55e572999050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59804/0 (socket says 172.21.15.91:59804) 2023-11-24T06:09:04.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.954+0000 7f0147fff700 1 -- 172.21.15.91:0/2917349016 learned_addr learned my addr 172.21.15.91:0/2917349016 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:04.959 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f0147fff700 1 -- 172.21.15.91:0/2917349016 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e572980ed0 msgr2=0x55e57297ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:04.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f0147fff700 1 --2- 172.21.15.91:0/2917349016 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e572980ed0 0x55e57297ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f0147fff700 1 -- 172.21.15.91:0/2917349016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e57296b080 msgr2=0x55e5729982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:04.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f0147fff700 1 --2- 172.21.15.91:0/2917349016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e57296b080 0x55e5729982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f0147fff700 1 -- 172.21.15.91:0/2917349016 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5726d0c10 con 0x55e572968490 2023-11-24T06:09:04.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f0147fff700 1 --2- 172.21.15.91:0/2917349016 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572968490 0x55e572999050 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x55e5729c34b0 tx=0x7f013c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988ba0438b614607 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:04.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f012e7fc700 1 -- 172.21.15.91:0/2917349016 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c015ce0 con 0x55e572968490 2023-11-24T06:09:04.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f012e7fc700 1 -- 172.21.15.91:0/2917349016 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013c01b040 con 0x55e572968490 2023-11-24T06:09:04.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f012e7fc700 1 -- 172.21.15.91:0/2917349016 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c0152d0 con 0x55e572968490 2023-11-24T06:09:04.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f015a4dfb80 1 -- 172.21.15.91:0/2917349016 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572968490 msgr2=0x55e572999050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/2917349016 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572968490 0x55e572999050 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x55e5729c34b0 tx=0x7f013c012000 comp rx=0 tx=0).stop 2023-11-24T06:09:04.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f015a4dfb80 1 -- 172.21.15.91:0/2917349016 shutdown_connections 2023-11-24T06:09:04.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/2917349016 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e572980ed0 0x55e57297ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/2917349016 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572968490 0x55e572999050 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/2917349016 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e57296b080 0x55e5729982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f015a4dfb80 1 -- 172.21.15.91:0/2917349016 >> 172.21.15.91:0/2917349016 conn(0x55e572899310 msgr2=0x55e572966d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:04.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.955+0000 7f015a4dfb80 1 -- 172.21.15.91:0/2917349016 shutdown_connections 2023-11-24T06:09:04.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.956+0000 7f015a4dfb80 1 -- 172.21.15.91:0/2917349016 wait complete. 2023-11-24T06:09:04.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.956+0000 7f015a4dfb80 1 -- start start 2023-11-24T06:09:04.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.956+0000 7f015a4dfb80 1 Processor -- start 2023-11-24T06:09:04.965 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f015a4dfb80 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572968490 0x55e5729c0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.965 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f015a4dfb80 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e57296b080 0x55e57297ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.965 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f015a4dfb80 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e572980ed0 0x55e5729a8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.965 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f015a4dfb80 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5729b3650 con 0x55e572968490 2023-11-24T06:09:04.965 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f015a4dfb80 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5729c1e50 con 0x55e572980ed0 2023-11-24T06:09:04.965 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f015a4dfb80 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e572a193a0 con 0x55e57296b080 2023-11-24T06:09:04.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f015a4dfb80 10 client.0 _ll_register_callbacks cb 0x7f0140009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-11-24T06:09:04.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f0147fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572968490 0x55e5729c0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f0147fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572968490 0x55e5729c0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59810/0 (socket says 172.21.15.91:59810) 2023-11-24T06:09:04.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f0147fff700 1 -- 172.21.15.91:0/1886316569 learned_addr learned my addr 172.21.15.91:0/1886316569 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:04.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f01477fe700 1 --2- 172.21.15.91:0/1886316569 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e57296b080 0x55e57297ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f014c883700 1 --2- 172.21.15.91:0/1886316569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e572980ed0 0x55e5729a8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f0147fff700 1 -- 172.21.15.91:0/1886316569 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e57296b080 msgr2=0x55e57297ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:04.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f0147fff700 1 --2- 172.21.15.91:0/1886316569 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e57296b080 0x55e57297ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f0147fff700 1 -- 172.21.15.91:0/1886316569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e572980ed0 msgr2=0x55e5729a8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:04.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f0147fff700 1 --2- 172.21.15.91:0/1886316569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e572980ed0 0x55e5729a8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.957+0000 7f0147fff700 1 -- 172.21.15.91:0/1886316569 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f013c00f040 con 0x55e572968490 2023-11-24T06:09:04.966 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f0147fff700 1 --2- 172.21.15.91:0/1886316569 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572968490 0x55e5729c0bd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f013c028010 tx=0x7f013c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f8ce95f3a28bcb7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:04.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f0126ffd700 10 client.0 ms_handle_connect on v2:172.21.15.91:3300/0 2023-11-24T06:09:04.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f0126ffd700 1 -- 172.21.15.91:0/1886316569 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c026190 con 0x55e572968490 2023-11-24T06:09:04.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f0126ffd700 1 -- 172.21.15.91:0/1886316569 <== mon.0 v2:172.21.15.91:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f013c0262f0 con 0x55e572968490 2023-11-24T06:09:04.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f015a4dfb80 1 -- 172.21.15.91:0/1886316569 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0138060670 con 0x55e572968490 2023-11-24T06:09:04.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f0126ffd700 1 -- 172.21.15.91:0/1886316569 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f013c0264f0 con 0x55e572968490 2023-11-24T06:09:04.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f015a4dfb80 10 client.5256 fetch_fsmap learned FSMap version 16 2023-11-24T06:09:04.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f015a4dfb80 1 -- 172.21.15.91:0/1886316569 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55e57270e200 con 0x55e572968490 2023-11-24T06:09:04.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f0126ffd700 1 -- 172.21.15.91:0/1886316569 <== mon.0 v2:172.21.15.91:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0138060670 con 0x55e572968490 2023-11-24T06:09:04.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f0126ffd700 1 -- 172.21.15.91:0/1886316569 <== mon.0 v2:172.21.15.91:3300/0 5 ==== fsmap(e 16) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f013c026650 con 0x55e572968490 2023-11-24T06:09:04.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f015a4dfb80 10 client.5256 fetch_fsmap finished waiting for FSMap version 16 2023-11-24T06:09:04.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f015a4dfb80 10 client.5256 resolve_mds: resolved name 'a' to daemon mds.a 2023-11-24T06:09:04.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f015a4dfb80 20 client.5256 populate_metadata read hostname 'smithi091' 2023-11-24T06:09:04.967 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/1886316569 >> [v2:172.21.15.91:6835/1301930708,v1:172.21.15.91:6837/1301930708] conn(0x55e572a18750 0x55e572a18b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:04.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f015a4dfb80 4 client.5256 mds_command: new command op to 5067 tid=0 multi_id=0 [{"prefix": "status"}] 2023-11-24T06:09:04.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f015a4dfb80 1 -- 172.21.15.91:0/1886316569 --> [v2:172.21.15.91:6835/1301930708,v1:172.21.15.91:6837/1301930708] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e572884f80 con 0x55e572a18750 2023-11-24T06:09:04.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f01477fe700 1 --2- 172.21.15.91:0/1886316569 >> [v2:172.21.15.91:6835/1301930708,v1:172.21.15.91:6837/1301930708] conn(0x55e572a18750 0x55e572a18b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:04.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.958+0000 7f0126ffd700 10 client.5256 ms_handle_connect on v2:172.21.15.91:6835/1301930708 2023-11-24T06:09:04.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.959+0000 7f01477fe700 1 --2- 172.21.15.91:0/1886316569 >> [v2:172.21.15.91:6835/1301930708,v1:172.21.15.91:6837/1301930708] conn(0x55e572a18750 0x55e572a18b30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=495bd93534b43a2a server_cookie=20d777c840949ccc in_seq=0 out_seq=0 2023-11-24T06:09:04.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.959+0000 7f0126ffd700 1 -- 172.21.15.91:0/1886316569 <== mds.0 v2:172.21.15.91:6835/1301930708 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55e572884f80 con 0x55e572a18750 2023-11-24T06:09:04.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.959+0000 7f0126ffd700 10 client.5256 handle_command_reply: tid=0 2023-11-24T06:09:04.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.959+0000 7f015a4dfb80 1 client.5256 shutdown 2023-11-24T06:09:04.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.959+0000 7f015a4dfb80 1 -- 172.21.15.91:0/1886316569 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572968490 msgr2=0x55e5729c0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.959+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/1886316569 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572968490 0x55e5729c0bd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f013c028010 tx=0x7f013c000ed0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.959+0000 7f015a4dfb80 1 -- 172.21.15.91:0/1886316569 shutdown_connections 2023-11-24T06:09:04.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.959+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/1886316569 >> [v2:172.21.15.91:6835/1301930708,v1:172.21.15.91:6837/1301930708] conn(0x55e572a18750 0x55e572a18b30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.968 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.959+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/1886316569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x55e572980ed0 0x55e5729a8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.959+0000 7f0126ffd700 0 client.5256 ms_handle_reset on v2:172.21.15.91:6835/1301930708 2023-11-24T06:09:04.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.959+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/1886316569 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x55e572968490 0x55e5729c0bd0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.959+0000 7f015a4dfb80 1 --2- 172.21.15.91:0/1886316569 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x55e57296b080 0x55e57297ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.959+0000 7f015a4dfb80 1 -- 172.21.15.91:0/1886316569 >> 172.21.15.91:0/1886316569 conn(0x55e572899310 msgr2=0x55e5729afa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:04.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.959+0000 7f015a4dfb80 1 -- 172.21.15.91:0/1886316569 shutdown_connections 2023-11-24T06:09:04.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.959+0000 7f015a4dfb80 1 -- 172.21.15.91:0/1886316569 wait complete. 2023-11-24T06:09:04.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.959+0000 7f015a4dfb80 20 client.5256 trim_cache size 0 max 16384 2023-11-24T06:09:04.969 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2023-11-24T06:09:04.969 INFO:teuthology.orchestra.run.smithi091.stdout: "cluster_fsid": "f7c07073-6cd6-496f-812a-c57c86a2fbbb", 2023-11-24T06:09:04.969 INFO:teuthology.orchestra.run.smithi091.stdout: "whoami": 0, 2023-11-24T06:09:04.970 INFO:teuthology.orchestra.run.smithi091.stdout: "id": 5067, 2023-11-24T06:09:04.970 INFO:teuthology.orchestra.run.smithi091.stdout: "want_state": "up:active", 2023-11-24T06:09:04.970 INFO:teuthology.orchestra.run.smithi091.stdout: "state": "up:active", 2023-11-24T06:09:04.970 INFO:teuthology.orchestra.run.smithi091.stdout: "fs_name": "cephfs", 2023-11-24T06:09:04.970 INFO:teuthology.orchestra.run.smithi091.stdout: "rank_uptime": 5.9163577719999996, 2023-11-24T06:09:04.970 INFO:teuthology.orchestra.run.smithi091.stdout: "mdsmap_epoch": 16, 2023-11-24T06:09:04.970 INFO:teuthology.orchestra.run.smithi091.stdout: "osdmap_epoch": 34, 2023-11-24T06:09:04.970 INFO:teuthology.orchestra.run.smithi091.stdout: "osdmap_epoch_barrier": 34, 2023-11-24T06:09:04.970 INFO:teuthology.orchestra.run.smithi091.stdout: "uptime": 14.835688522 2023-11-24T06:09:04.970 INFO:teuthology.orchestra.run.smithi091.stdout:} 2023-11-24T06:09:04.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.960+0000 7f014e2e6700 1 -- 172.21.15.91:0/2537283817 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f013406aed0 msgr2=0x7f013406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.960+0000 7f014e2e6700 1 --2- 172.21.15.91:0/2537283817 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f013406aed0 0x7f013406d390 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f013c000f60 tx=0x7f013c007be0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.960+0000 7f014e2e6700 1 -- 172.21.15.91:0/2537283817 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0148143ae0 msgr2=0x7f01480cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:04.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.960+0000 7f014e2e6700 1 --2- 172.21.15.91:0/2537283817 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0148143ae0 0x7f01480cba40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f013800c750 tx=0x7f013800cb10 comp rx=0 tx=0).stop 2023-11-24T06:09:04.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.960+0000 7f014e2e6700 1 -- 172.21.15.91:0/2537283817 shutdown_connections 2023-11-24T06:09:04.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.960+0000 7f014e2e6700 1 --2- 172.21.15.91:0/2537283817 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f013406aed0 0x7f013406d390 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.960+0000 7f014e2e6700 1 --2- 172.21.15.91:0/2537283817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f01480722c0 0x7f01481395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.960+0000 7f014e2e6700 1 --2- 172.21.15.91:0/2537283817 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f01481440f0 0x7f01480cf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.960+0000 7f014e2e6700 1 --2- 172.21.15.91:0/2537283817 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0148143ae0 0x7f01480cba40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:04.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.960+0000 7f014e2e6700 1 -- 172.21.15.91:0/2537283817 >> 172.21.15.91:0/2537283817 conn(0x7f01480b2740 msgr2=0x7f014806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:04.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.960+0000 7f014e2e6700 1 -- 172.21.15.91:0/2537283817 shutdown_connections 2023-11-24T06:09:04.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:04.960+0000 7f014e2e6700 1 -- 172.21.15.91:0/2537283817 wait complete. 2023-11-24T06:09:04.974 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T06:09:05.095 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.092+0000 7eff99b2c700 1 Processor -- start 2023-11-24T06:09:05.095 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.092+0000 7eff99b2c700 1 -- start start 2023-11-24T06:09:05.098 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.093+0000 7eff99b2c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7eff94142810 0x7eff9406afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:05.098 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.093+0000 7eff99b2c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff941459a0 0x7eff9406b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:05.098 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.093+0000 7eff99b2c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7eff9406e580 0x7eff94071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:05.098 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.093+0000 7eff99b2c700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff940cf5f0 con 0x7eff94142810 2023-11-24T06:09:05.098 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.093+0000 7eff99b2c700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff9406ea80 con 0x7eff9406e580 2023-11-24T06:09:05.098 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.093+0000 7eff99b2c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9406ed60 con 0x7eff941459a0 2023-11-24T06:09:05.099 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.093+0000 7eff937fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7eff94142810 0x7eff9406afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:05.099 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.094+0000 7eff937fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7eff94142810 0x7eff9406afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59818/0 (socket says 172.21.15.91:59818) 2023-11-24T06:09:05.099 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.094+0000 7eff92ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff941459a0 0x7eff9406b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:05.099 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.094+0000 7eff937fe700 1 -- 172.21.15.91:0/3448733861 learned_addr learned my addr 172.21.15.91:0/3448733861 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:05.099 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.094+0000 7eff93fff700 1 --2- 172.21.15.91:0/3448733861 >> v2:172.21.15.94:3300/0 conn(0x7eff9406e580 0x7eff94071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:05.099 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.094+0000 7eff937fe700 1 -- 172.21.15.91:0/3448733861 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff941459a0 msgr2=0x7eff9406b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:05.099 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.094+0000 7eff937fe700 1 --2- 172.21.15.91:0/3448733861 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff941459a0 0x7eff9406b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:05.099 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.094+0000 7eff937fe700 1 -- 172.21.15.91:0/3448733861 >> v2:172.21.15.94:3300/0 conn(0x7eff9406e580 msgr2=0x7eff94071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:05.099 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.094+0000 7eff937fe700 1 --2- 172.21.15.91:0/3448733861 >> v2:172.21.15.94:3300/0 conn(0x7eff9406e580 0x7eff94071bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:05.099 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.094+0000 7eff937fe700 1 -- 172.21.15.91:0/3448733861 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9406f040 con 0x7eff94142810 2023-11-24T06:09:05.099 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.094+0000 7eff937fe700 1 --2- 172.21.15.91:0/3448733861 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff94142810 0x7eff9406afd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7eff940679e0 tx=0x7eff84009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d4347810357a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:05.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.094+0000 7eff927fc700 1 -- 172.21.15.91:0/3448733861 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84011830 con 0x7eff94142810 2023-11-24T06:09:05.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.094+0000 7eff927fc700 1 -- 172.21.15.91:0/3448733861 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff84011990 con 0x7eff94142810 2023-11-24T06:09:05.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.094+0000 7eff927fc700 1 -- 172.21.15.91:0/3448733861 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84011d20 con 0x7eff94142810 2023-11-24T06:09:05.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.095+0000 7eff99b2c700 1 -- 172.21.15.91:0/3448733861 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff94142810 msgr2=0x7eff9406afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:05.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.095+0000 7eff99b2c700 1 --2- 172.21.15.91:0/3448733861 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff94142810 0x7eff9406afd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7eff940679e0 tx=0x7eff84009bd0 comp rx=0 tx=0).stop 2023-11-24T06:09:05.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.095+0000 7eff99b2c700 1 -- 172.21.15.91:0/3448733861 shutdown_connections 2023-11-24T06:09:05.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.095+0000 7eff99b2c700 1 --2- 172.21.15.91:0/3448733861 >> v2:172.21.15.94:3300/0 conn(0x7eff9406e580 0x7eff94071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:05.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.095+0000 7eff99b2c700 1 --2- 172.21.15.91:0/3448733861 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff94142810 0x7eff9406afd0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:05.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.095+0000 7eff99b2c700 1 --2- 172.21.15.91:0/3448733861 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff941459a0 0x7eff9406b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:05.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.095+0000 7eff99b2c700 1 -- 172.21.15.91:0/3448733861 >> 172.21.15.91:0/3448733861 conn(0x7eff940b2740 msgr2=0x7eff940b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:05.100 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.095+0000 7eff99b2c700 1 -- 172.21.15.91:0/3448733861 shutdown_connections 2023-11-24T06:09:05.101 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.095+0000 7eff99b2c700 1 -- 172.21.15.91:0/3448733861 wait complete. 2023-11-24T06:09:05.103 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.096+0000 7eff99b2c700 1 Processor -- start 2023-11-24T06:09:05.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.096+0000 7eff99b2c700 1 -- start start 2023-11-24T06:09:05.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.097+0000 7eff99b2c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff9406e580 0x7eff941f0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:05.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.097+0000 7eff99b2c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff94142810 0x7eff940a9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:05.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.097+0000 7eff99b2c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff941459a0 0x7eff941347a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:05.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.097+0000 7eff99b2c700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff941f12e0 con 0x7eff9406e580 2023-11-24T06:09:05.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.097+0000 7eff99b2c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff94131740 con 0x7eff941459a0 2023-11-24T06:09:05.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.097+0000 7eff99b2c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff94131a20 con 0x7eff94142810 2023-11-24T06:09:05.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.097+0000 7eff937fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff9406e580 0x7eff941f0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:05.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.097+0000 7eff937fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff9406e580 0x7eff941f0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59824/0 (socket says 172.21.15.91:59824) 2023-11-24T06:09:05.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.097+0000 7eff937fe700 1 -- 172.21.15.91:0/1920209864 learned_addr learned my addr 172.21.15.91:0/1920209864 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:05.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.097+0000 7eff93fff700 1 --2- 172.21.15.91:0/1920209864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff941459a0 0x7eff941347a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:05.104 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.097+0000 7eff92ffd700 1 --2- 172.21.15.91:0/1920209864 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff94142810 0x7eff940a9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:05.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.097+0000 7eff937fe700 1 -- 172.21.15.91:0/1920209864 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff94142810 msgr2=0x7eff940a9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:05.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.098+0000 7eff937fe700 1 --2- 172.21.15.91:0/1920209864 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff94142810 0x7eff940a9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:05.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.098+0000 7eff937fe700 1 -- 172.21.15.91:0/1920209864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff941459a0 msgr2=0x7eff941347a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:05.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.098+0000 7eff937fe700 1 --2- 172.21.15.91:0/1920209864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff941459a0 0x7eff941347a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:05.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.098+0000 7eff937fe700 1 -- 172.21.15.91:0/1920209864 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff84009550 con 0x7eff9406e580 2023-11-24T06:09:05.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.098+0000 7eff93fff700 1 --2- 172.21.15.91:0/1920209864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff941459a0 0x7eff941347a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:05.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.098+0000 7eff92ffd700 1 --2- 172.21.15.91:0/1920209864 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff94142810 0x7eff940a9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:05.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.098+0000 7eff937fe700 1 --2- 172.21.15.91:0/1920209864 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff9406e580 0x7eff941f0da0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7eff84000c00 tx=0x7eff84004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:05.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.098+0000 7eff7bfff700 1 -- 172.21.15.91:0/1920209864 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84004300 con 0x7eff9406e580 2023-11-24T06:09:05.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.098+0000 7eff7bfff700 1 -- 172.21.15.91:0/1920209864 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8401f070 con 0x7eff9406e580 2023-11-24T06:09:05.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.098+0000 7eff7bfff700 1 -- 172.21.15.91:0/1920209864 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff840036a0 con 0x7eff9406e580 2023-11-24T06:09:05.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.098+0000 7eff99b2c700 1 -- 172.21.15.91:0/1920209864 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff94131d00 con 0x7eff9406e580 2023-11-24T06:09:05.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.098+0000 7eff99b2c700 1 -- 172.21.15.91:0/1920209864 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff941322c0 con 0x7eff9406e580 2023-11-24T06:09:05.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.099+0000 7eff99b2c700 1 -- 172.21.15.91:0/1920209864 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff94132450 con 0x7eff9406e580 2023-11-24T06:09:05.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.099+0000 7eff7bfff700 1 -- 172.21.15.91:0/1920209864 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7eff84021030 con 0x7eff9406e580 2023-11-24T06:09:05.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.100+0000 7eff7bfff700 1 --2- 172.21.15.91:0/1920209864 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7eff8006aed0 0x7eff8006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:05.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.100+0000 7eff7bfff700 1 -- 172.21.15.91:0/1920209864 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff8408a7c0 con 0x7eff9406e580 2023-11-24T06:09:05.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.100+0000 7eff92ffd700 1 --2- 172.21.15.91:0/1920209864 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7eff8006aed0 0x7eff8006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:05.106 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.100+0000 7eff92ffd700 1 --2- 172.21.15.91:0/1920209864 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7eff8006aed0 0x7eff8006d390 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7eff941f4960 tx=0x7eff88008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:05.107 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.104+0000 7eff7bfff700 1 -- 172.21.15.91:0/1920209864 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff84055900 con 0x7eff9406e580 2023-11-24T06:09:05.340 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.337+0000 7eff99b2c700 1 -- 172.21.15.91:0/1920209864 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7eff94068670 con 0x7eff9406e580 2023-11-24T06:09:05.340 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.337+0000 7eff7bfff700 1 -- 172.21.15.91:0/1920209864 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7eff84015070 con 0x7eff9406e580 2023-11-24T06:09:05.340 INFO:teuthology.orchestra.run.smithi091.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T06:09:05.342 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.339+0000 7eff99b2c700 1 -- 172.21.15.91:0/1920209864 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7eff8006aed0 msgr2=0x7eff8006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:05.343 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.339+0000 7eff99b2c700 1 --2- 172.21.15.91:0/1920209864 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7eff8006aed0 0x7eff8006d390 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7eff941f4960 tx=0x7eff88008040 comp rx=0 tx=0).stop 2023-11-24T06:09:05.343 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.339+0000 7eff99b2c700 1 -- 172.21.15.91:0/1920209864 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff9406e580 msgr2=0x7eff941f0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:05.343 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.339+0000 7eff99b2c700 1 --2- 172.21.15.91:0/1920209864 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff9406e580 0x7eff941f0da0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7eff84000c00 tx=0x7eff84004750 comp rx=0 tx=0).stop 2023-11-24T06:09:05.343 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.339+0000 7eff99b2c700 1 -- 172.21.15.91:0/1920209864 shutdown_connections 2023-11-24T06:09:05.343 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.339+0000 7eff99b2c700 1 --2- 172.21.15.91:0/1920209864 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7eff8006aed0 0x7eff8006d390 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:05.343 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.339+0000 7eff99b2c700 1 --2- 172.21.15.91:0/1920209864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff941459a0 0x7eff941347a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:05.343 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.339+0000 7eff99b2c700 1 --2- 172.21.15.91:0/1920209864 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff9406e580 0x7eff941f0da0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:05.343 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.339+0000 7eff99b2c700 1 --2- 172.21.15.91:0/1920209864 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff94142810 0x7eff940a9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:05.343 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.339+0000 7eff99b2c700 1 -- 172.21.15.91:0/1920209864 >> 172.21.15.91:0/1920209864 conn(0x7eff940b2740 msgr2=0x7eff940b1970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:05.343 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.340+0000 7eff99b2c700 1 -- 172.21.15.91:0/1920209864 shutdown_connections 2023-11-24T06:09:05.343 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:05.340+0000 7eff99b2c700 1 -- 172.21.15.91:0/1920209864 wait complete. 2023-11-24T06:09:05.354 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T06:09:05.354 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T06:09:05.354 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi091.front.sepia.ceph.com 2023-11-24T06:09:05.354 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-11-24T06:09:05.354 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-11-24T06:09:05.355 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T06:09:05.355 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T06:09:05.355 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T06:09:05.355 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T06:09:05.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:05.355 DEBUG:teuthology.orchestra.run.smithi091:> ip addr 2023-11-24T06:09:05.372 INFO:teuthology.orchestra.run.smithi091.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:09:05.372 INFO:teuthology.orchestra.run.smithi091.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:09:05.373 INFO:teuthology.orchestra.run.smithi091.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:09:05.373 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:09:05.373 INFO:teuthology.orchestra.run.smithi091.stdout: inet6 ::1/128 scope host 2023-11-24T06:09:05.373 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:09:05.373 INFO:teuthology.orchestra.run.smithi091.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:09:05.373 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 0c:c4:7a:d8:d9:c4 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:09:05.373 INFO:teuthology.orchestra.run.smithi091.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:09:05.373 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 0c:c4:7a:d8:d9:c5 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:09:05.373 INFO:teuthology.orchestra.run.smithi091.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:09:05.373 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 0c:c4:7a:8f:d3:d4 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:09:05.373 INFO:teuthology.orchestra.run.smithi091.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T06:09:05.373 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 0c:c4:7a:8f:d3:d5 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:09:05.373 INFO:teuthology.orchestra.run.smithi091.stdout: inet 172.21.15.91/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T06:09:05.373 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft 42034sec preferred_lft 42034sec 2023-11-24T06:09:05.374 INFO:teuthology.orchestra.run.smithi091.stdout: inet6 fe80::f6f7:b098:fe54:3959/64 scope link noprefixroute 2023-11-24T06:09:05.374 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:09:05.374 INFO:teuthology.orchestra.run.smithi091.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:09:05.374 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 36:7a:21:0e:c4:94 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:09:05.374 INFO:teuthology.orchestra.run.smithi091.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T06:09:05.374 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:09:05.374 INFO:teuthology.orchestra.run.smithi091.stdout: inet6 fe80::b813:36ff:fe84:b31a/64 scope link 2023-11-24T06:09:05.374 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:09:05.374 INFO:teuthology.orchestra.run.smithi091.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:09:05.374 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 76:69:41:d1:ea:d2 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T06:09:05.374 INFO:teuthology.orchestra.run.smithi091.stdout: inet6 fe80::7469:41ff:fed1:ead2/64 scope link 2023-11-24T06:09:05.374 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:09:05.374 INFO:teuthology.orchestra.run.smithi091.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:09:05.374 INFO:teuthology.orchestra.run.smithi091.stdout: link/ether 36:7a:21:0e:c4:94 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T06:09:05.375 INFO:teuthology.orchestra.run.smithi091.stdout: inet6 fe80::347a:21ff:fe0e:c494/64 scope link 2023-11-24T06:09:05.375 INFO:teuthology.orchestra.run.smithi091.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:09:05.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:05.375 DEBUG:teuthology.orchestra.run.smithi091:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T06:09:05.437 INFO:teuthology.orchestra.run.smithi091.stdout:1 2023-11-24T06:09:05.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:05.438 DEBUG:teuthology.orchestra.run.smithi091:> route 2023-11-24T06:09:05.498 INFO:teuthology.orchestra.run.smithi091.stdout:Kernel IP routing table 2023-11-24T06:09:05.499 INFO:teuthology.orchestra.run.smithi091.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T06:09:05.499 INFO:teuthology.orchestra.run.smithi091.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T06:09:05.499 INFO:teuthology.orchestra.run.smithi091.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T06:09:05.499 INFO:teuthology.orchestra.run.smithi091.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T06:09:05.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:05.499 DEBUG:teuthology.orchestra.run.smithi091:> (cd / && exec bash -c ' 2023-11-24T06:09:05.499 DEBUG:teuthology.orchestra.run.smithi091:> set -e 2023-11-24T06:09:05.499 DEBUG:teuthology.orchestra.run.smithi091:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T06:09:05.499 DEBUG:teuthology.orchestra.run.smithi091:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T06:09:05.500 DEBUG:teuthology.orchestra.run.smithi091:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T06:09:05.500 DEBUG:teuthology.orchestra.run.smithi091:> ') 2023-11-24T06:09:05.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:05.604 DEBUG:teuthology.orchestra.run.smithi091:> ip netns list 2023-11-24T06:09:05.620 INFO:teuthology.orchestra.run.smithi091.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T06:09:05.620 INFO:teuthology.orchestra.run.smithi091.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T06:09:05.620 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-11-24T06:09:05.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:05.621 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip link set brx.0 up 2023-11-24T06:09:05.687 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:09:05.688 DEBUG:teuthology.orchestra.run.smithi091:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-11-24T06:09:05.743 INFO:teuthology.orchestra.run.smithi091.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-11-24T06:09:05.743 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:09:05.743 DEBUG:teuthology.orchestra.run.smithi091:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-11-24T06:09:05.800 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-11-24T06:09:05.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:05.800 DEBUG:teuthology.orchestra.run.smithi091:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:09:05.884 INFO:teuthology.orchestra.run.smithi091.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:09:05.884 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:09:05.907 INFO:teuthology.orchestra.run.smithi091.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:09:05.907 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:09:05.907 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: resolved to: "172.21.15.91:3300,172.21.15.94:3300,172.21.15.94:3301" 2023-11-24T06:09:05.908 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: trying mount with new device syntax: 0@f7c07073-6cd6-496f-812a-c57c86a2fbbb.cephfs=/ 2023-11-24T06:09:05.908 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.91:3300/172.21.15.94:3300/172.21.15.94:3301" will pass to kernel 2023-11-24T06:09:05.908 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: v2 mount support check returned 2 2023-11-24T06:09:05.908 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: kernel does not support v2 syntax 2023-11-24T06:09:05.908 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:09:05.937 INFO:teuthology.orchestra.run.smithi091.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:09:05.937 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:09:05.937 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: resolved to: "172.21.15.91:3300,172.21.15.94:3300,172.21.15.94:3301" 2023-11-24T06:09:05.938 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: trying mount with new device syntax: 0@f7c07073-6cd6-496f-812a-c57c86a2fbbb.cephfs=/ 2023-11-24T06:09:05.938 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.91:3300/172.21.15.94:3300/172.21.15.94:3301" will pass to kernel 2023-11-24T06:09:05.938 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: v2 mount support check returned 2 2023-11-24T06:09:05.938 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: kernel does not support v2 syntax 2023-11-24T06:09:05.938 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:09:05.938 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: resolved to: "172.21.15.91:6789,172.21.15.94:6789,172.21.15.94:6790" 2023-11-24T06:09:05.938 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.91:6789,172.21.15.94:6789,172.21.15.94:6790:/ 2023-11-24T06:09:05.938 INFO:teuthology.orchestra.run.smithi091.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,key=0,mds_namespace=cephfs,fsid=f7c07073-6cd6-496f-812a-c57c86a2fbbb" will pass to kernel 2023-11-24T06:09:05.946 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T06:09:05.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:05.947 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-11-24T06:09:05.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:05.979 DEBUG:teuthology.orchestra.run.smithi091:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:09:06.060 INFO:teuthology.orchestra.run.smithi091.stdout:client5265 2023-11-24T06:09:06.060 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:09:06.060 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:09:06.118 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-11-24T06:09:06.277 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.275+0000 7f3081d99700 1 Processor -- start 2023-11-24T06:09:06.278 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.275+0000 7f3081d99700 1 -- start start 2023-11-24T06:09:06.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.275+0000 7f3081d99700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f307c15bde0 0x7f307c15c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:06.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.276+0000 7f3081d99700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c0d0a90 con 0x7f307c0ce5c0 2023-11-24T06:09:06.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.276+0000 7f3081d99700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c06ee10 con 0x7f307c0cf2b0 2023-11-24T06:09:06.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.276+0000 7f3081d99700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f307c06f0f0 con 0x7f307c15bde0 2023-11-24T06:09:06.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.276+0000 7f307b7fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f307c15bde0 0x7f307c15c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:06.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.276+0000 7f307b7fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f307c15bde0 0x7f307c15c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59836/0 (socket says 172.21.15.91:59836) 2023-11-24T06:09:06.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.276+0000 7f307b7fe700 1 -- 172.21.15.91:0/3529236068 learned_addr learned my addr 172.21.15.91:0/3529236068 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:06.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.276+0000 7f307b7fe700 1 -- 172.21.15.91:0/3529236068 >> v1:172.21.15.94:6789/0 conn(0x7f307c0cf2b0 legacy=0x7f307c0d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:06.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.276+0000 7f307b7fe700 1 -- 172.21.15.91:0/3529236068 >> v1:172.21.15.91:6789/0 conn(0x7f307c0ce5c0 legacy=0x7f307c06fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:06.280 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.276+0000 7f307b7fe700 1 -- 172.21.15.91:0/3529236068 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c06f3d0 con 0x7f307c15bde0 2023-11-24T06:09:06.281 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.276+0000 7f307a7fc700 1 -- 172.21.15.91:0/3529236068 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111761916 0 0) 0x7f307c0d0a90 con 0x7f307c0ce5c0 2023-11-24T06:09:06.281 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.276+0000 7f307b7fe700 1 --2- 172.21.15.91:0/3529236068 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f307c15bde0 0x7f307c15c1c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f306c0098f0 tx=0x7f306c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55825eaaee223cbd server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:06.281 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.276+0000 7f307a7fc700 1 -- 172.21.15.91:0/3529236068 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c004230 con 0x7f307c15bde0 2023-11-24T06:09:06.281 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.276+0000 7f307a7fc700 1 -- 172.21.15.91:0/3529236068 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306c004390 con 0x7f307c15bde0 2023-11-24T06:09:06.281 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.276+0000 7f307a7fc700 1 -- 172.21.15.91:0/3529236068 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c0036a0 con 0x7f307c15bde0 2023-11-24T06:09:06.281 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.277+0000 7f3081d99700 1 -- 172.21.15.91:0/3529236068 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f307c15bde0 msgr2=0x7f307c15c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:06.281 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.277+0000 7f3081d99700 1 --2- 172.21.15.91:0/3529236068 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f307c15bde0 0x7f307c15c1c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f306c0098f0 tx=0x7f306c009c00 comp rx=0 tx=0).stop 2023-11-24T06:09:06.281 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.277+0000 7f3081d99700 1 -- 172.21.15.91:0/3529236068 shutdown_connections 2023-11-24T06:09:06.281 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.277+0000 7f3081d99700 1 --2- 172.21.15.91:0/3529236068 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f307c15bde0 0x7f307c15c1c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:06.281 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.277+0000 7f3081d99700 1 -- 172.21.15.91:0/3529236068 >> 172.21.15.91:0/3529236068 conn(0x7f307c0b3720 msgr2=0x7f307c0b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:06.281 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.277+0000 7f3081d99700 1 -- 172.21.15.91:0/3529236068 shutdown_connections 2023-11-24T06:09:06.281 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.277+0000 7f3081d99700 1 -- 172.21.15.91:0/3529236068 wait complete. 2023-11-24T06:09:06.284 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.278+0000 7f3081d99700 1 Processor -- start 2023-11-24T06:09:06.284 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.278+0000 7f3081d99700 1 -- start start 2023-11-24T06:09:06.284 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.279+0000 7f3081d99700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f307c0ce5c0 0x7f307c0b7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:06.284 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.279+0000 7f3081d99700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f307c0cf2b0 0x7f307c0c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:06.284 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.279+0000 7f3081d99700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f307c15bde0 0x7f307c149000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:06.284 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.279+0000 7f3081d99700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c0b61d0 con 0x7f307c0cf2b0 2023-11-24T06:09:06.285 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.279+0000 7f3081d99700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c0b64b0 con 0x7f307c0ce5c0 2023-11-24T06:09:06.285 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.279+0000 7f3081d99700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307c0b6790 con 0x7f307c15bde0 2023-11-24T06:09:06.285 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.279+0000 7f307affd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f307c0cf2b0 0x7f307c0c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:06.285 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.279+0000 7f307affd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f307c0cf2b0 0x7f307c0c76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59842/0 (socket says 172.21.15.91:59842) 2023-11-24T06:09:06.285 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.279+0000 7f307affd700 1 -- 172.21.15.91:0/4191487911 learned_addr learned my addr 172.21.15.91:0/4191487911 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:06.285 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.279+0000 7f307b7fe700 1 --2- 172.21.15.91:0/4191487911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f307c0ce5c0 0x7f307c0b7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:06.285 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.279+0000 7f307bfff700 1 --2- 172.21.15.91:0/4191487911 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f307c15bde0 0x7f307c149000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:06.285 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.279+0000 7f307affd700 1 -- 172.21.15.91:0/4191487911 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f307c15bde0 msgr2=0x7f307c149000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:06.285 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.279+0000 7f307affd700 1 --2- 172.21.15.91:0/4191487911 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f307c15bde0 0x7f307c149000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:06.285 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.279+0000 7f307affd700 1 -- 172.21.15.91:0/4191487911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f307c0ce5c0 msgr2=0x7f307c0b7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:06.285 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.279+0000 7f307affd700 1 --2- 172.21.15.91:0/4191487911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f307c0ce5c0 0x7f307c0b7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:06.285 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.279+0000 7f307affd700 1 -- 172.21.15.91:0/4191487911 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306c009550 con 0x7f307c0cf2b0 2023-11-24T06:09:06.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.280+0000 7f307affd700 1 --2- 172.21.15.91:0/4191487911 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f307c0cf2b0 0x7f307c0c76a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f307000c4a0 tx=0x7f307000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:06.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.280+0000 7f3063fff700 1 -- 172.21.15.91:0/4191487911 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3070011790 con 0x7f307c0cf2b0 2023-11-24T06:09:06.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.280+0000 7f3063fff700 1 -- 172.21.15.91:0/4191487911 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30700118f0 con 0x7f307c0cf2b0 2023-11-24T06:09:06.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.280+0000 7f3081d99700 1 -- 172.21.15.91:0/4191487911 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307c0b6a70 con 0x7f307c0cf2b0 2023-11-24T06:09:06.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.280+0000 7f3063fff700 1 -- 172.21.15.91:0/4191487911 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3070011ca0 con 0x7f307c0cf2b0 2023-11-24T06:09:06.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.280+0000 7f3081d99700 1 -- 172.21.15.91:0/4191487911 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307c0b77b0 con 0x7f307c0cf2b0 2023-11-24T06:09:06.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.281+0000 7f3081d99700 1 -- 172.21.15.91:0/4191487911 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307c068670 con 0x7f307c0cf2b0 2023-11-24T06:09:06.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.281+0000 7f3063fff700 1 -- 172.21.15.91:0/4191487911 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f307001c460 con 0x7f307c0cf2b0 2023-11-24T06:09:06.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.281+0000 7f3063fff700 1 --2- 172.21.15.91:0/4191487911 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f306806aed0 0x7f306806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:06.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.282+0000 7f3063fff700 1 -- 172.21.15.91:0/4191487911 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3070089250 con 0x7f307c0cf2b0 2023-11-24T06:09:06.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.282+0000 7f307b7fe700 1 --2- 172.21.15.91:0/4191487911 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f306806aed0 0x7f306806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:06.287 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.282+0000 7f307b7fe700 1 --2- 172.21.15.91:0/4191487911 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f306806aed0 0x7f306806d390 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f306c009130 tx=0x7f306c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:06.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.286+0000 7f3063fff700 1 -- 172.21.15.91:0/4191487911 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3070054300 con 0x7f307c0cf2b0 2023-11-24T06:09:06.524 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.521+0000 7f3081d99700 1 -- 172.21.15.91:0/4191487911 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f307c1e4c20 con 0x7f307c0cf2b0 2023-11-24T06:09:06.524 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.521+0000 7f3063fff700 1 -- 172.21.15.91:0/4191487911 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3070057920 con 0x7f307c0cf2b0 2023-11-24T06:09:06.524 INFO:teuthology.orchestra.run.smithi091.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-11-24T06:09:06.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.524+0000 7f3081d99700 1 -- 172.21.15.91:0/4191487911 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f306806aed0 msgr2=0x7f306806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:06.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.524+0000 7f3081d99700 1 --2- 172.21.15.91:0/4191487911 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f306806aed0 0x7f306806d390 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f306c009130 tx=0x7f306c008030 comp rx=0 tx=0).stop 2023-11-24T06:09:06.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.524+0000 7f3081d99700 1 -- 172.21.15.91:0/4191487911 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f307c0cf2b0 msgr2=0x7f307c0c76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:06.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.524+0000 7f3081d99700 1 --2- 172.21.15.91:0/4191487911 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f307c0cf2b0 0x7f307c0c76a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f307000c4a0 tx=0x7f307000c7b0 comp rx=0 tx=0).stop 2023-11-24T06:09:06.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.524+0000 7f3081d99700 1 -- 172.21.15.91:0/4191487911 shutdown_connections 2023-11-24T06:09:06.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.524+0000 7f3081d99700 1 --2- 172.21.15.91:0/4191487911 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f306806aed0 0x7f306806d390 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:06.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.524+0000 7f3081d99700 1 --2- 172.21.15.91:0/4191487911 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f307c0ce5c0 0x7f307c0b7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:06.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.524+0000 7f3081d99700 1 --2- 172.21.15.91:0/4191487911 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f307c15bde0 0x7f307c149000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:06.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.524+0000 7f3081d99700 1 --2- 172.21.15.91:0/4191487911 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f307c0cf2b0 0x7f307c0c76a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:06.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.524+0000 7f3081d99700 1 -- 172.21.15.91:0/4191487911 >> 172.21.15.91:0/4191487911 conn(0x7f307c0b3720 msgr2=0x7f307c15b010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:06.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.524+0000 7f3081d99700 1 -- 172.21.15.91:0/4191487911 shutdown_connections 2023-11-24T06:09:06.528 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:06.524+0000 7f3081d99700 1 -- 172.21.15.91:0/4191487911 wait complete. 2023-11-24T06:09:06.538 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-11-24T06:09:06.539 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-11-24T06:09:06.539 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi094.front.sepia.ceph.com 2023-11-24T06:09:06.539 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-11-24T06:09:06.539 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-11-24T06:09:06.539 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-11-24T06:09:06.539 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-11-24T06:09:06.539 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-11-24T06:09:06.539 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T06:09:06.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:06.539 DEBUG:teuthology.orchestra.run.smithi094:> ip addr 2023-11-24T06:09:06.561 INFO:teuthology.orchestra.run.smithi094.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-11-24T06:09:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-11-24T06:09:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: inet 127.0.0.1/8 scope host lo 2023-11-24T06:09:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:09:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 ::1/128 scope host 2023-11-24T06:09:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:09:06.561 INFO:teuthology.orchestra.run.smithi094.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:09:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4c brd ff:ff:ff:ff:ff:ff 2023-11-24T06:09:06.561 INFO:teuthology.orchestra.run.smithi094.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:09:06.561 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:d8:db:4d brd ff:ff:ff:ff:ff:ff 2023-11-24T06:09:06.561 INFO:teuthology.orchestra.run.smithi094.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-11-24T06:09:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:62 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:09:06.562 INFO:teuthology.orchestra.run.smithi094.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-11-24T06:09:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0c:c4:7a:8f:d7:63 brd ff:ff:ff:ff:ff:ff 2023-11-24T06:09:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: inet 172.21.15.94/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-11-24T06:09:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft 42028sec preferred_lft 42028sec 2023-11-24T06:09:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 fe80::d54a:432d:e3af:656/64 scope link noprefixroute 2023-11-24T06:09:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:09:06.562 INFO:teuthology.orchestra.run.smithi094.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-11-24T06:09:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 02:89:29:f0:b8:7c brd ff:ff:ff:ff:ff:ff 2023-11-24T06:09:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-11-24T06:09:06.562 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:09:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 fe80::84ef:82ff:fe13:f3ab/64 scope link 2023-11-24T06:09:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:09:06.563 INFO:teuthology.orchestra.run.smithi094.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:09:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 0a:12:45:d5:24:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T06:09:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 fe80::812:45ff:fed5:2496/64 scope link 2023-11-24T06:09:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:09:06.563 INFO:teuthology.orchestra.run.smithi094.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-11-24T06:09:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: link/ether 02:89:29:f0:b8:7c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T06:09:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: inet6 fe80::89:29ff:fef0:b87c/64 scope link 2023-11-24T06:09:06.563 INFO:teuthology.orchestra.run.smithi094.stdout: valid_lft forever preferred_lft forever 2023-11-24T06:09:06.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:06.564 DEBUG:teuthology.orchestra.run.smithi094:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-11-24T06:09:06.630 INFO:teuthology.orchestra.run.smithi094.stdout:1 2023-11-24T06:09:06.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:06.632 DEBUG:teuthology.orchestra.run.smithi094:> route 2023-11-24T06:09:06.691 INFO:teuthology.orchestra.run.smithi094.stdout:Kernel IP routing table 2023-11-24T06:09:06.692 INFO:teuthology.orchestra.run.smithi094.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-11-24T06:09:06.692 INFO:teuthology.orchestra.run.smithi094.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-11-24T06:09:06.692 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-11-24T06:09:06.692 INFO:teuthology.orchestra.run.smithi094.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-11-24T06:09:06.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:06.693 DEBUG:teuthology.orchestra.run.smithi094:> (cd / && exec bash -c ' 2023-11-24T06:09:06.693 DEBUG:teuthology.orchestra.run.smithi094:> set -e 2023-11-24T06:09:06.693 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-11-24T06:09:06.693 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-11-24T06:09:06.693 DEBUG:teuthology.orchestra.run.smithi094:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-11-24T06:09:06.693 DEBUG:teuthology.orchestra.run.smithi094:> ') 2023-11-24T06:09:06.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:06.803 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2023-11-24T06:09:06.820 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T06:09:06.820 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T06:09:06.820 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2023-11-24T06:09:06.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:06.821 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link set brx.0 up 2023-11-24T06:09:06.888 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:09:06.889 DEBUG:teuthology.orchestra.run.smithi094:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-11-24T06:09:06.946 INFO:teuthology.orchestra.run.smithi094.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-11-24T06:09:06.946 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-11-24T06:09:06.946 DEBUG:teuthology.orchestra.run.smithi094:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-11-24T06:09:07.002 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2023-11-24T06:09:07.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:07.003 DEBUG:teuthology.orchestra.run.smithi094:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:09:07.089 INFO:teuthology.orchestra.run.smithi094.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:09:07.090 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:09:07.114 INFO:teuthology.orchestra.run.smithi094.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:09:07.114 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:09:07.114 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: resolved to: "172.21.15.91:3300,172.21.15.94:3300,172.21.15.94:3301" 2023-11-24T06:09:07.114 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: trying mount with new device syntax: 1@f7c07073-6cd6-496f-812a-c57c86a2fbbb.cephfs=/ 2023-11-24T06:09:07.115 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.91:3300/172.21.15.94:3300/172.21.15.94:3301" will pass to kernel 2023-11-24T06:09:07.115 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: v2 mount support check returned 2 2023-11-24T06:09:07.115 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: kernel does not support v2 syntax 2023-11-24T06:09:07.115 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:09:07.157 INFO:teuthology.orchestra.run.smithi094.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-11-24T06:09:07.157 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-11-24T06:09:07.157 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: resolved to: "172.21.15.91:3300,172.21.15.94:3300,172.21.15.94:3301" 2023-11-24T06:09:07.157 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: trying mount with new device syntax: 1@f7c07073-6cd6-496f-812a-c57c86a2fbbb.cephfs=/ 2023-11-24T06:09:07.157 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.91:3300/172.21.15.94:3300/172.21.15.94:3301" will pass to kernel 2023-11-24T06:09:07.157 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: v2 mount support check returned 2 2023-11-24T06:09:07.157 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: kernel does not support v2 syntax 2023-11-24T06:09:07.157 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: switching to using v1 address with old syntax 2023-11-24T06:09:07.157 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: resolved to: "172.21.15.91:6789,172.21.15.94:6789,172.21.15.94:6790" 2023-11-24T06:09:07.157 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.91:6789,172.21.15.94:6789,172.21.15.94:6790:/ 2023-11-24T06:09:07.157 INFO:teuthology.orchestra.run.smithi094.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,key=1,mds_namespace=cephfs,fsid=f7c07073-6cd6-496f-812a-c57c86a2fbbb" will pass to kernel 2023-11-24T06:09:07.174 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-11-24T06:09:07.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:07.174 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-11-24T06:09:07.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:07.201 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-11-24T06:09:07.274 INFO:teuthology.orchestra.run.smithi094.stdout:client4999 2023-11-24T06:09:07.274 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:09:07.274 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:09:07.330 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-11-24T06:09:07.469 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.466+0000 7fd00151c700 1 Processor -- start 2023-11-24T06:09:07.470 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.467+0000 7fd00151c700 1 -- start start 2023-11-24T06:09:07.471 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.467+0000 7fd00151c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcffc15f2c0 0x7fcffc0ada70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:07.471 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.467+0000 7fd00151c700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcffc0731c0 con 0x7fcffc155060 2023-11-24T06:09:07.471 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.467+0000 7fd00151c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcffc15e820 con 0x7fcffc06ec40 2023-11-24T06:09:07.471 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.467+0000 7fd00151c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc15e630 con 0x7fcffc15f2c0 2023-11-24T06:09:07.471 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.468+0000 7fcffaffd700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fcffc155060 0x7fcffc072600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54782/0 (socket says 172.21.15.91:54782) 2023-11-24T06:09:07.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.468+0000 7fcffaffd700 1 -- 172.21.15.91:0/4157198217 learned_addr learned my addr 172.21.15.91:0/4157198217 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:07.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.468+0000 7fcffa7fc700 1 --2- 172.21.15.91:0/4157198217 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcffc15f2c0 0x7fcffc0ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:07.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.468+0000 7fcff9ffb700 1 -- 172.21.15.91:0/4157198217 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47912639 0 0) 0x7fcffc0731c0 con 0x7fcffc155060 2023-11-24T06:09:07.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.468+0000 7fcff9ffb700 1 -- 172.21.15.91:0/4157198217 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfe40035b0 con 0x7fcffc155060 2023-11-24T06:09:07.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.468+0000 7fcff9ffb700 1 -- 172.21.15.91:0/4157198217 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3811987874 0 0) 0x7fcfe40035b0 con 0x7fcffc155060 2023-11-24T06:09:07.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.468+0000 7fcffa7fc700 1 -- 172.21.15.91:0/4157198217 >> v1:172.21.15.94:6789/0 conn(0x7fcffc06ec40 legacy=0x7fcffc070ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:07.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.468+0000 7fcffa7fc700 1 -- 172.21.15.91:0/4157198217 >> v1:172.21.15.91:6789/0 conn(0x7fcffc155060 legacy=0x7fcffc072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:07.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.468+0000 7fcffa7fc700 1 -- 172.21.15.91:0/4157198217 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc072ee0 con 0x7fcffc15f2c0 2023-11-24T06:09:07.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.468+0000 7fcffa7fc700 1 --2- 172.21.15.91:0/4157198217 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcffc15f2c0 0x7fcffc0ada70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009870 tx=0x7fcfec009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8b2302b922af9b7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:07.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.468+0000 7fcff9ffb700 1 -- 172.21.15.91:0/4157198217 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0001670 con 0x7fcffc15f2c0 2023-11-24T06:09:07.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.468+0000 7fcff9ffb700 1 -- 172.21.15.91:0/4157198217 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfec004030 con 0x7fcffc15f2c0 2023-11-24T06:09:07.472 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.468+0000 7fcff9ffb700 1 -- 172.21.15.91:0/4157198217 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfec0043c0 con 0x7fcffc15f2c0 2023-11-24T06:09:07.473 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.469+0000 7fd00151c700 1 -- 172.21.15.91:0/4157198217 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcffc15f2c0 msgr2=0x7fcffc0ada70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:07.473 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.469+0000 7fd00151c700 1 --2- 172.21.15.91:0/4157198217 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcffc15f2c0 0x7fcffc0ada70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009870 tx=0x7fcfec009b80 comp rx=0 tx=0).stop 2023-11-24T06:09:07.473 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.469+0000 7fd00151c700 1 -- 172.21.15.91:0/4157198217 shutdown_connections 2023-11-24T06:09:07.473 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.469+0000 7fd00151c700 1 --2- 172.21.15.91:0/4157198217 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcffc15f2c0 0x7fcffc0ada70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:07.474 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.469+0000 7fd00151c700 1 -- 172.21.15.91:0/4157198217 >> 172.21.15.91:0/4157198217 conn(0x7fcffc0cca10 msgr2=0x7fcffc0cce20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:07.474 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.469+0000 7fd00151c700 1 -- 172.21.15.91:0/4157198217 shutdown_connections 2023-11-24T06:09:07.474 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.469+0000 7fd00151c700 1 -- 172.21.15.91:0/4157198217 wait complete. 2023-11-24T06:09:07.474 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.470+0000 7fd00151c700 1 Processor -- start 2023-11-24T06:09:07.474 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.470+0000 7fd00151c700 1 -- start start 2023-11-24T06:09:07.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.471+0000 7fd00151c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcffc06ec40 0x7fcffc144ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:07.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.471+0000 7fd00151c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcffc155060 0x7fcffc1f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:07.476 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.471+0000 7fd00151c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcffc15f2c0 0x7fcffc0cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:07.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.471+0000 7fd00151c700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc13f120 con 0x7fcffc06ec40 2023-11-24T06:09:07.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.471+0000 7fd00151c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc145790 con 0x7fcffc155060 2023-11-24T06:09:07.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.471+0000 7fd00151c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc161e00 con 0x7fcffc15f2c0 2023-11-24T06:09:07.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.471+0000 7fcffaffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcffc06ec40 0x7fcffc144ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:07.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.471+0000 7fcffaffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcffc06ec40 0x7fcffc144ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59854/0 (socket says 172.21.15.91:59854) 2023-11-24T06:09:07.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.471+0000 7fcffaffd700 1 -- 172.21.15.91:0/2351961685 learned_addr learned my addr 172.21.15.91:0/2351961685 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:07.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.471+0000 7fcffa7fc700 1 --2- 172.21.15.91:0/2351961685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcffc155060 0x7fcffc1f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:07.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.471+0000 7fcffb7fe700 1 --2- 172.21.15.91:0/2351961685 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcffc15f2c0 0x7fcffc0cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:07.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.472+0000 7fcffaffd700 1 -- 172.21.15.91:0/2351961685 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcffc15f2c0 msgr2=0x7fcffc0cbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:07.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.472+0000 7fcffaffd700 1 --2- 172.21.15.91:0/2351961685 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcffc15f2c0 0x7fcffc0cbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:07.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.472+0000 7fcffaffd700 1 -- 172.21.15.91:0/2351961685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcffc155060 msgr2=0x7fcffc1f79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:07.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.472+0000 7fcffaffd700 1 --2- 172.21.15.91:0/2351961685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcffc155060 0x7fcffc1f79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:07.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.472+0000 7fcffaffd700 1 -- 172.21.15.91:0/2351961685 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfec0094a0 con 0x7fcffc06ec40 2023-11-24T06:09:07.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.472+0000 7fcffaffd700 1 --2- 172.21.15.91:0/2351961685 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcffc06ec40 0x7fcffc144ed0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcff000c700 tx=0x7fcff000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:07.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.472+0000 7fcfe37fe700 1 -- 172.21.15.91:0/2351961685 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0004c80 con 0x7fcffc06ec40 2023-11-24T06:09:07.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.472+0000 7fcfe37fe700 1 -- 172.21.15.91:0/2351961685 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff0004e00 con 0x7fcffc06ec40 2023-11-24T06:09:07.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.472+0000 7fd00151c700 1 -- 172.21.15.91:0/2351961685 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcffc1620e0 con 0x7fcffc06ec40 2023-11-24T06:09:07.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.472+0000 7fcfe37fe700 1 -- 172.21.15.91:0/2351961685 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0007310 con 0x7fcffc06ec40 2023-11-24T06:09:07.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.472+0000 7fd00151c700 1 -- 172.21.15.91:0/2351961685 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcffc141610 con 0x7fcffc06ec40 2023-11-24T06:09:07.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.473+0000 7fd00151c700 1 -- 172.21.15.91:0/2351961685 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcffc141470 con 0x7fcffc06ec40 2023-11-24T06:09:07.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.473+0000 7fcfe37fe700 1 -- 172.21.15.91:0/2351961685 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcff0007470 con 0x7fcffc06ec40 2023-11-24T06:09:07.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.474+0000 7fcfe37fe700 1 --2- 172.21.15.91:0/2351961685 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcfe806aed0 0x7fcfe806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:07.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.474+0000 7fcfe37fe700 1 -- 172.21.15.91:0/2351961685 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcff0088ab0 con 0x7fcffc06ec40 2023-11-24T06:09:07.479 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.474+0000 7fcffa7fc700 1 --2- 172.21.15.91:0/2351961685 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcfe806aed0 0x7fcfe806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:07.479 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.474+0000 7fcffa7fc700 1 --2- 172.21.15.91:0/2351961685 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcfe806aed0 0x7fcfe806d390 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcfec0097c0 tx=0x7fcfec008000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:07.481 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.478+0000 7fcfe37fe700 1 -- 172.21.15.91:0/2351961685 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcffc141470 con 0x7fcffc06ec40 2023-11-24T06:09:07.714 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:07.711+0000 7fd00151c700 1 -- 172.21.15.91:0/2351961685 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcffc141470 con 0x7fcffc06ec40 2023-11-24T06:09:08.105 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.102+0000 7fcfe37fe700 1 -- 172.21.15.91:0/2351961685 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 136+0+0 (secure 0 0 0) 0x7fcffc141470 con 0x7fcffc06ec40 2023-11-24T06:09:08.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.105+0000 7fd00151c700 1 -- 172.21.15.91:0/2351961685 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcfe806aed0 msgr2=0x7fcfe806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.105+0000 7fd00151c700 1 --2- 172.21.15.91:0/2351961685 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcfe806aed0 0x7fcfe806d390 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcfec0097c0 tx=0x7fcfec008000 comp rx=0 tx=0).stop 2023-11-24T06:09:08.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.105+0000 7fd00151c700 1 -- 172.21.15.91:0/2351961685 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcffc06ec40 msgr2=0x7fcffc144ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.105+0000 7fd00151c700 1 --2- 172.21.15.91:0/2351961685 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcffc06ec40 0x7fcffc144ed0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcff000c700 tx=0x7fcff000cac0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.105+0000 7fd00151c700 1 -- 172.21.15.91:0/2351961685 shutdown_connections 2023-11-24T06:09:08.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.105+0000 7fd00151c700 1 --2- 172.21.15.91:0/2351961685 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fcfe806aed0 0x7fcfe806d390 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.105+0000 7fd00151c700 1 --2- 172.21.15.91:0/2351961685 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fcffc15f2c0 0x7fcffc0cbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.108 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.105+0000 7fd00151c700 1 --2- 172.21.15.91:0/2351961685 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fcffc06ec40 0x7fcffc144ed0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.105+0000 7fd00151c700 1 --2- 172.21.15.91:0/2351961685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fcffc155060 0x7fcffc1f79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.105+0000 7fd00151c700 1 -- 172.21.15.91:0/2351961685 >> 172.21.15.91:0/2351961685 conn(0x7fcffc0cca10 msgr2=0x7fcffc0afa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:08.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.105+0000 7fd00151c700 1 -- 172.21.15.91:0/2351961685 shutdown_connections 2023-11-24T06:09:08.109 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.105+0000 7fd00151c700 1 -- 172.21.15.91:0/2351961685 wait complete. 2023-11-24T06:09:08.120 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-11-24T06:09:08.242 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.239+0000 7f687cb08700 1 Processor -- start 2023-11-24T06:09:08.242 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.239+0000 7f687cb08700 1 -- start start 2023-11-24T06:09:08.244 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f687cb08700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f687806b620 0x7f68780b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.244 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f687cb08700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f687806ee60 0x7f68780b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.244 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f687cb08700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6878071f50 con 0x7f68780b5ab0 2023-11-24T06:09:08.244 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f687cb08700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6878072230 con 0x7f687806ee60 2023-11-24T06:09:08.244 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f687cb08700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6878072510 con 0x7f687806b620 2023-11-24T06:09:08.244 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f6876d9d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f687806ee60 0x7f68780b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.245 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f6876d9d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f687806ee60 0x7f68780b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59866/0 (socket says 172.21.15.91:59866) 2023-11-24T06:09:08.245 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f6876d9d700 1 -- 172.21.15.91:0/2434238044 learned_addr learned my addr 172.21.15.91:0/2434238044 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:08.245 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f6875d9b700 1 --1- 172.21.15.91:0/2434238044 >> v1:172.21.15.91:6789/0 conn(0x7f68780b5ab0 0x7f6878161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54796/0 (socket says 172.21.15.91:54796) 2023-11-24T06:09:08.245 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f687659c700 1 --2- 172.21.15.91:0/2434238044 >> v2:172.21.15.94:3300/0 conn(0x7f687806b620 0x7f68780b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.245 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f6876d9d700 1 -- 172.21.15.91:0/2434238044 >> v2:172.21.15.94:3300/0 conn(0x7f687806b620 msgr2=0x7f68780b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:08.245 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f6876d9d700 1 --2- 172.21.15.91:0/2434238044 >> v2:172.21.15.94:3300/0 conn(0x7f687806b620 0x7f68780b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.245 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f6876d9d700 1 -- 172.21.15.91:0/2434238044 >> v1:172.21.15.91:6789/0 conn(0x7f68780b5ab0 legacy=0x7f6878161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.245 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f6876d9d700 1 -- 172.21.15.91:0/2434238044 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68780727f0 con 0x7f687806ee60 2023-11-24T06:09:08.245 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f687559a700 1 -- 172.21.15.91:0/2434238044 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128566140 0 0) 0x7f6878071f50 con 0x7f68780b5ab0 2023-11-24T06:09:08.245 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f6876d9d700 1 --2- 172.21.15.91:0/2434238044 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f687806ee60 0x7f68780b3f80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f687806d2e0 tx=0x7f687000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81d48d5bcbc8ffaa server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:08.245 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.240+0000 7f687559a700 1 -- 172.21.15.91:0/2434238044 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687000e070 con 0x7f687806ee60 2023-11-24T06:09:08.245 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.241+0000 7f687559a700 1 -- 172.21.15.91:0/2434238044 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687000b9c0 con 0x7f687806ee60 2023-11-24T06:09:08.245 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.241+0000 7f687559a700 1 -- 172.21.15.91:0/2434238044 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687000bd70 con 0x7f687806ee60 2023-11-24T06:09:08.246 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.241+0000 7f687cb08700 1 -- 172.21.15.91:0/2434238044 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f687806ee60 msgr2=0x7f68780b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.246 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.241+0000 7f687cb08700 1 --2- 172.21.15.91:0/2434238044 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f687806ee60 0x7f68780b3f80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f687806d2e0 tx=0x7f687000b330 comp rx=0 tx=0).stop 2023-11-24T06:09:08.246 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.241+0000 7f687cb08700 1 -- 172.21.15.91:0/2434238044 shutdown_connections 2023-11-24T06:09:08.246 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.241+0000 7f687cb08700 1 --2- 172.21.15.91:0/2434238044 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f687806ee60 0x7f68780b3f80 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.246 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.241+0000 7f687cb08700 1 --2- 172.21.15.91:0/2434238044 >> v2:172.21.15.94:3300/0 conn(0x7f687806b620 0x7f68780b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.246 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.241+0000 7f687cb08700 1 -- 172.21.15.91:0/2434238044 >> 172.21.15.91:0/2434238044 conn(0x7f68780b2740 msgr2=0x7f68780b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:08.246 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.241+0000 7f687cb08700 1 -- 172.21.15.91:0/2434238044 shutdown_connections 2023-11-24T06:09:08.246 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.241+0000 7f687cb08700 1 -- 172.21.15.91:0/2434238044 wait complete. 2023-11-24T06:09:08.249 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.243+0000 7f687cb08700 1 Processor -- start 2023-11-24T06:09:08.249 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.243+0000 7f687cb08700 1 -- start start 2023-11-24T06:09:08.249 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.243+0000 7f687cb08700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f687806b620 0x7f68780b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.249 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.243+0000 7f687cb08700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f687806ee60 0x7f6878143f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.249 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.243+0000 7f687cb08700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68780b5ab0 0x7f68781444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.249 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.243+0000 7f687cb08700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68780c68c0 con 0x7f68780b5ab0 2023-11-24T06:09:08.249 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.243+0000 7f687cb08700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68780cbf00 con 0x7f687806ee60 2023-11-24T06:09:08.250 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.243+0000 7f687cb08700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68781449f0 con 0x7f687806b620 2023-11-24T06:09:08.250 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.243+0000 7f6876d9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68780b5ab0 0x7f68781444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.250 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.243+0000 7f6876d9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68780b5ab0 0x7f68781444b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59872/0 (socket says 172.21.15.91:59872) 2023-11-24T06:09:08.250 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.243+0000 7f6876d9d700 1 -- 172.21.15.91:0/2725610961 learned_addr learned my addr 172.21.15.91:0/2725610961 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:08.250 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.244+0000 7f687659c700 1 --2- 172.21.15.91:0/2725610961 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f687806b620 0x7f68780b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.250 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.244+0000 7f6875d9b700 1 --2- 172.21.15.91:0/2725610961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f687806ee60 0x7f6878143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.250 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.244+0000 7f6876d9d700 1 -- 172.21.15.91:0/2725610961 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f687806b620 msgr2=0x7f68780b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.250 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.244+0000 7f6876d9d700 1 --2- 172.21.15.91:0/2725610961 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f687806b620 0x7f68780b3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.250 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.244+0000 7f6876d9d700 1 -- 172.21.15.91:0/2725610961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f687806ee60 msgr2=0x7f6878143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.250 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.244+0000 7f6876d9d700 1 --2- 172.21.15.91:0/2725610961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f687806ee60 0x7f6878143f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.250 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.244+0000 7f6876d9d700 1 -- 172.21.15.91:0/2725610961 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6870009b30 con 0x7f68780b5ab0 2023-11-24T06:09:08.250 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.244+0000 7f6875d9b700 1 --2- 172.21.15.91:0/2725610961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f687806ee60 0x7f6878143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:08.251 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.244+0000 7f6876d9d700 1 --2- 172.21.15.91:0/2725610961 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68780b5ab0 0x7f68781444b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f68780d1bd0 tx=0x7f6870004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:08.251 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.244+0000 7f6866ffd700 1 -- 172.21.15.91:0/2725610961 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68700092e0 con 0x7f68780b5ab0 2023-11-24T06:09:08.251 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.244+0000 7f6866ffd700 1 -- 172.21.15.91:0/2725610961 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687001c440 con 0x7f68780b5ab0 2023-11-24T06:09:08.251 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.244+0000 7f687cb08700 1 -- 172.21.15.91:0/2725610961 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68780c9ff0 con 0x7f68780b5ab0 2023-11-24T06:09:08.251 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.244+0000 7f6866ffd700 1 -- 172.21.15.91:0/2725610961 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687001c7f0 con 0x7f68780b5ab0 2023-11-24T06:09:08.251 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.244+0000 7f687cb08700 1 -- 172.21.15.91:0/2725610961 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68780ca890 con 0x7f68780b5ab0 2023-11-24T06:09:08.251 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.245+0000 7f687cb08700 1 -- 172.21.15.91:0/2725610961 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68780caa50 con 0x7f68780b5ab0 2023-11-24T06:09:08.251 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.245+0000 7f6866ffd700 1 -- 172.21.15.91:0/2725610961 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f687001c950 con 0x7f68780b5ab0 2023-11-24T06:09:08.251 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.246+0000 7f6866ffd700 1 --2- 172.21.15.91:0/2725610961 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f685806aed0 0x7f685806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.251 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.246+0000 7f6866ffd700 1 -- 172.21.15.91:0/2725610961 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f687008af60 con 0x7f68780b5ab0 2023-11-24T06:09:08.251 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.246+0000 7f687659c700 1 --2- 172.21.15.91:0/2725610961 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f685806aed0 0x7f685806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.251 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.246+0000 7f687659c700 1 --2- 172.21.15.91:0/2725610961 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f685806aed0 0x7f685806d390 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f68780c7a10 tx=0x7f6868008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:08.253 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.250+0000 7f6866ffd700 1 -- 172.21.15.91:0/2725610961 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68700560a0 con 0x7f68780b5ab0 2023-11-24T06:09:08.473 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.470+0000 7f687cb08700 1 -- 172.21.15.91:0/2725610961 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f687806ba00 con 0x7f685806aed0 2023-11-24T06:09:08.474 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.472+0000 7f6866ffd700 1 -- 172.21.15.91:0/2725610961 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6868009180 con 0x7f685806aed0 2023-11-24T06:09:08.474 INFO:teuthology.orchestra.run.smithi091.stdout:[ 2023-11-24T06:09:08.474 INFO:teuthology.orchestra.run.smithi091.stdout: { 2023-11-24T06:09:08.474 INFO:teuthology.orchestra.run.smithi091.stdout: "name": "cephfs" 2023-11-24T06:09:08.475 INFO:teuthology.orchestra.run.smithi091.stdout: } 2023-11-24T06:09:08.475 INFO:teuthology.orchestra.run.smithi091.stdout:] 2023-11-24T06:09:08.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.474+0000 7f687cb08700 1 -- 172.21.15.91:0/2725610961 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f685806aed0 msgr2=0x7f685806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.474+0000 7f687cb08700 1 --2- 172.21.15.91:0/2725610961 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f685806aed0 0x7f685806d390 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f68780c7a10 tx=0x7f6868008040 comp rx=0 tx=0).stop 2023-11-24T06:09:08.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.474+0000 7f687cb08700 1 -- 172.21.15.91:0/2725610961 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68780b5ab0 msgr2=0x7f68781444b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.474+0000 7f687cb08700 1 --2- 172.21.15.91:0/2725610961 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68780b5ab0 0x7f68781444b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f68780d1bd0 tx=0x7f6870004ad0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.474+0000 7f687cb08700 1 -- 172.21.15.91:0/2725610961 shutdown_connections 2023-11-24T06:09:08.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.474+0000 7f687cb08700 1 --2- 172.21.15.91:0/2725610961 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f685806aed0 0x7f685806d390 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.474+0000 7f687cb08700 1 --2- 172.21.15.91:0/2725610961 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f68780b5ab0 0x7f68781444b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.474+0000 7f687cb08700 1 --2- 172.21.15.91:0/2725610961 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f687806b620 0x7f68780b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.477 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.474+0000 7f687cb08700 1 --2- 172.21.15.91:0/2725610961 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f687806ee60 0x7f6878143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.474+0000 7f687cb08700 1 -- 172.21.15.91:0/2725610961 >> 172.21.15.91:0/2725610961 conn(0x7f68780b2740 msgr2=0x7f68780af150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:08.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.474+0000 7f687cb08700 1 -- 172.21.15.91:0/2725610961 shutdown_connections 2023-11-24T06:09:08.478 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.474+0000 7f687cb08700 1 -- 172.21.15.91:0/2725610961 wait complete. 2023-11-24T06:09:08.488 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-11-24T06:09:08.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.608+0000 7f95c8060700 1 Processor -- start 2023-11-24T06:09:08.611 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.608+0000 7f95c8060700 1 -- start start 2023-11-24T06:09:08.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.609+0000 7f95c8060700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f95c00b47e0 0x7f95c00cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.609+0000 7f95c8060700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c00cffc0 con 0x7f95c01616f0 2023-11-24T06:09:08.613 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.609+0000 7f95c8060700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c00d2c70 con 0x7f95c0161ad0 2023-11-24T06:09:08.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.609+0000 7f95c8060700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95c00d0f20 con 0x7f95c00b47e0 2023-11-24T06:09:08.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.609+0000 7f95c65fd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f95c00b47e0 0x7f95c00cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.609+0000 7f95c5dfc700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f95c01616f0 0x7f95c00d1d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54810/0 (socket says 172.21.15.91:54810) 2023-11-24T06:09:08.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.609+0000 7f95c5dfc700 1 -- 172.21.15.91:0/3632696647 learned_addr learned my addr 172.21.15.91:0/3632696647 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:08.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.610+0000 7f95c65fd700 1 -- 172.21.15.91:0/3632696647 >> v1:172.21.15.94:6789/0 conn(0x7f95c0161ad0 legacy=0x7f95c00d2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.610+0000 7f95c65fd700 1 -- 172.21.15.91:0/3632696647 >> v1:172.21.15.91:6789/0 conn(0x7f95c01616f0 legacy=0x7f95c00d1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.610+0000 7f95c4dfa700 1 -- 172.21.15.91:0/3632696647 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 884464998 0 0) 0x7f95b0001110 con 0x7f95c01616f0 2023-11-24T06:09:08.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.610+0000 7f95c65fd700 1 -- 172.21.15.91:0/3632696647 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c00d1200 con 0x7f95c00b47e0 2023-11-24T06:09:08.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.610+0000 7f95c65fd700 1 --2- 172.21.15.91:0/3632696647 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95c00b47e0 0x7f95c00cfa80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009d90 tx=0x7f95b800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85fd76ceef1299d2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:08.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.610+0000 7f95c4dfa700 1 -- 172.21.15.91:0/3632696647 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b800e070 con 0x7f95c00b47e0 2023-11-24T06:09:08.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.610+0000 7f95c4dfa700 1 -- 172.21.15.91:0/3632696647 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b800bbd0 con 0x7f95c00b47e0 2023-11-24T06:09:08.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.610+0000 7f95c4dfa700 1 -- 172.21.15.91:0/3632696647 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8012430 con 0x7f95c00b47e0 2023-11-24T06:09:08.614 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.610+0000 7f95c8060700 1 -- 172.21.15.91:0/3632696647 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95c00b47e0 msgr2=0x7f95c00cfa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.610+0000 7f95c8060700 1 --2- 172.21.15.91:0/3632696647 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95c00b47e0 0x7f95c00cfa80 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009d90 tx=0x7f95b800b330 comp rx=0 tx=0).stop 2023-11-24T06:09:08.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.610+0000 7f95c8060700 1 -- 172.21.15.91:0/3632696647 shutdown_connections 2023-11-24T06:09:08.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.610+0000 7f95c8060700 1 --2- 172.21.15.91:0/3632696647 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95c00b47e0 0x7f95c00cfa80 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.610+0000 7f95c8060700 1 -- 172.21.15.91:0/3632696647 >> 172.21.15.91:0/3632696647 conn(0x7f95c00d3e60 msgr2=0x7f95c00d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:08.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.611+0000 7f95c8060700 1 -- 172.21.15.91:0/3632696647 shutdown_connections 2023-11-24T06:09:08.615 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.611+0000 7f95c8060700 1 -- 172.21.15.91:0/3632696647 wait complete. 2023-11-24T06:09:08.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.612+0000 7f95c8060700 1 Processor -- start 2023-11-24T06:09:08.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.612+0000 7f95c8060700 1 -- start start 2023-11-24T06:09:08.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.612+0000 7f95c8060700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95c00b47e0 0x7f95c014d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.612+0000 7f95c8060700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f95c01616f0 0x7f95c00c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.612+0000 7f95c8060700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95c0161ad0 0x7f95c00c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.612+0000 7f95c8060700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c0154da0 con 0x7f95c00b47e0 2023-11-24T06:09:08.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.612+0000 7f95c8060700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c00ca380 con 0x7f95c0161ad0 2023-11-24T06:09:08.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.612+0000 7f95c8060700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c00ca660 con 0x7f95c01616f0 2023-11-24T06:09:08.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.613+0000 7f95c5dfc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95c00b47e0 0x7f95c014d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.613+0000 7f95c5dfc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95c00b47e0 0x7f95c014d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59882/0 (socket says 172.21.15.91:59882) 2023-11-24T06:09:08.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.613+0000 7f95c5dfc700 1 -- 172.21.15.91:0/700053948 learned_addr learned my addr 172.21.15.91:0/700053948 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:08.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.613+0000 7f95c65fd700 1 --2- 172.21.15.91:0/700053948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95c0161ad0 0x7f95c00c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.613+0000 7f95c55fb700 1 --2- 172.21.15.91:0/700053948 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f95c01616f0 0x7f95c00c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.613+0000 7f95c5dfc700 1 -- 172.21.15.91:0/700053948 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f95c01616f0 msgr2=0x7f95c00c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.613+0000 7f95c5dfc700 1 --2- 172.21.15.91:0/700053948 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f95c01616f0 0x7f95c00c97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.613+0000 7f95c5dfc700 1 -- 172.21.15.91:0/700053948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95c0161ad0 msgr2=0x7f95c00c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.613+0000 7f95c5dfc700 1 --2- 172.21.15.91:0/700053948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95c0161ad0 0x7f95c00c9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.613+0000 7f95c5dfc700 1 -- 172.21.15.91:0/700053948 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b8009a70 con 0x7f95c00b47e0 2023-11-24T06:09:08.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.613+0000 7f95c5dfc700 1 --2- 172.21.15.91:0/700053948 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95c00b47e0 0x7f95c014d300 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f95b000c520 tx=0x7f95b000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:08.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.613+0000 7f95b67fc700 1 -- 172.21.15.91:0/700053948 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b0004c80 con 0x7f95c00b47e0 2023-11-24T06:09:08.619 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.613+0000 7f95b67fc700 1 -- 172.21.15.91:0/700053948 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b0004e20 con 0x7f95c00b47e0 2023-11-24T06:09:08.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.613+0000 7f95c8060700 1 -- 172.21.15.91:0/700053948 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c01546e0 con 0x7f95c00b47e0 2023-11-24T06:09:08.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.613+0000 7f95b67fc700 1 -- 172.21.15.91:0/700053948 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b0007310 con 0x7f95c00b47e0 2023-11-24T06:09:08.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.613+0000 7f95c8060700 1 -- 172.21.15.91:0/700053948 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c00b2740 con 0x7f95c00b47e0 2023-11-24T06:09:08.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.614+0000 7f95c8060700 1 -- 172.21.15.91:0/700053948 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c0155010 con 0x7f95c00b47e0 2023-11-24T06:09:08.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.615+0000 7f95b67fc700 1 -- 172.21.15.91:0/700053948 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f95b0007470 con 0x7f95c00b47e0 2023-11-24T06:09:08.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.615+0000 7f95b67fc700 1 --2- 172.21.15.91:0/700053948 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f95a406aed0 0x7f95a406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.615+0000 7f95b67fc700 1 -- 172.21.15.91:0/700053948 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f95b0089230 con 0x7f95c00b47e0 2023-11-24T06:09:08.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.615+0000 7f95c55fb700 1 --2- 172.21.15.91:0/700053948 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f95a406aed0 0x7f95a406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.620 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.615+0000 7f95c55fb700 1 --2- 172.21.15.91:0/700053948 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f95a406aed0 0x7f95a406d390 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f95c01537c0 tx=0x7f95bc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:08.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.619+0000 7f95b67fc700 1 -- 172.21.15.91:0/700053948 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f95b00542e0 con 0x7f95c00b47e0 2023-11-24T06:09:08.812 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.809+0000 7f95c8060700 1 -- 172.21.15.91:0/700053948 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f95c0068480 con 0x7f95c00b47e0 2023-11-24T06:09:08.812 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.809+0000 7f95b67fc700 1 -- 172.21.15.91:0/700053948 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f95b0007dd0 con 0x7f95c00b47e0 2023-11-24T06:09:08.814 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.811+0000 7f95c8060700 1 -- 172.21.15.91:0/700053948 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f95a406aed0 msgr2=0x7f95a406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.811+0000 7f95c8060700 1 --2- 172.21.15.91:0/700053948 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f95a406aed0 0x7f95a406d390 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f95c01537c0 tx=0x7f95bc009040 comp rx=0 tx=0).stop 2023-11-24T06:09:08.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.811+0000 7f95c8060700 1 -- 172.21.15.91:0/700053948 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95c00b47e0 msgr2=0x7f95c014d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.811+0000 7f95c8060700 1 --2- 172.21.15.91:0/700053948 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95c00b47e0 0x7f95c014d300 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f95b000c520 tx=0x7f95b000c8e0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.811+0000 7f95c8060700 1 -- 172.21.15.91:0/700053948 shutdown_connections 2023-11-24T06:09:08.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.811+0000 7f95c8060700 1 --2- 172.21.15.91:0/700053948 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f95a406aed0 0x7f95a406d390 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.811+0000 7f95c8060700 1 --2- 172.21.15.91:0/700053948 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f95c0161ad0 0x7f95c00c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.811+0000 7f95c8060700 1 --2- 172.21.15.91:0/700053948 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f95c00b47e0 0x7f95c014d300 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.811+0000 7f95c8060700 1 --2- 172.21.15.91:0/700053948 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f95c01616f0 0x7f95c00c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.811+0000 7f95c8060700 1 -- 172.21.15.91:0/700053948 >> 172.21.15.91:0/700053948 conn(0x7f95c00d3e60 msgr2=0x7f95c00d0160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:08.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.812+0000 7f95c8060700 1 -- 172.21.15.91:0/700053948 shutdown_connections 2023-11-24T06:09:08.815 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.812+0000 7f95c8060700 1 -- 172.21.15.91:0/700053948 wait complete. 2023-11-24T06:09:08.826 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol4424 --mode=777 2023-11-24T06:09:08.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.948+0000 7f2772360700 1 Processor -- start 2023-11-24T06:09:08.951 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.948+0000 7f2772360700 1 -- start start 2023-11-24T06:09:08.953 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.949+0000 7f2772360700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f276c06afd0 0x7f276c0b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.953 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.949+0000 7f2772360700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f276c06b3b0 0x7f276c15e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.953 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.949+0000 7f2772360700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f276c071ec0 con 0x7f276c072350 2023-11-24T06:09:08.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.949+0000 7f2772360700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f276c071b90 con 0x7f276c06afd0 2023-11-24T06:09:08.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.949+0000 7f2772360700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f276c15ed30 con 0x7f276c06b3b0 2023-11-24T06:09:08.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.949+0000 7f276bfff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f276c06afd0 0x7f276c0b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.949+0000 7f276bfff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f276c06afd0 0x7f276c0b59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59890/0 (socket says 172.21.15.91:59890) 2023-11-24T06:09:08.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.949+0000 7f27708fd700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f276c072350 0x7f276c06dbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54828/0 (socket says 172.21.15.91:54828) 2023-11-24T06:09:08.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.949+0000 7f276bfff700 1 -- 172.21.15.91:0/2345461054 learned_addr learned my addr 172.21.15.91:0/2345461054 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:08.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.949+0000 7f276b7fe700 1 --2- 172.21.15.91:0/2345461054 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f276c06b3b0 0x7f276c15e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.949+0000 7f276bfff700 1 -- 172.21.15.91:0/2345461054 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f276c06b3b0 msgr2=0x7f276c15e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:08.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.949+0000 7f276bfff700 1 --2- 172.21.15.91:0/2345461054 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f276c06b3b0 0x7f276c15e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.949+0000 7f276bfff700 1 -- 172.21.15.91:0/2345461054 >> v1:172.21.15.91:6789/0 conn(0x7f276c072350 legacy=0x7f276c06dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.949+0000 7f276bfff700 1 -- 172.21.15.91:0/2345461054 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f276c15f010 con 0x7f276c06afd0 2023-11-24T06:09:08.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.949+0000 7f276affd700 1 -- 172.21.15.91:0/2345461054 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922963961 0 0) 0x7f276c071ec0 con 0x7f276c072350 2023-11-24T06:09:08.954 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.949+0000 7f276bfff700 1 --2- 172.21.15.91:0/2345461054 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f276c06afd0 0x7f276c0b59d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f27600098f0 tx=0x7f2760009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2abe68264a563c20 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:08.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.950+0000 7f276affd700 1 -- 172.21.15.91:0/2345461054 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276000b8a0 con 0x7f276c06afd0 2023-11-24T06:09:08.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.950+0000 7f276affd700 1 -- 172.21.15.91:0/2345461054 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276000ba00 con 0x7f276c06afd0 2023-11-24T06:09:08.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.950+0000 7f276affd700 1 -- 172.21.15.91:0/2345461054 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276000bd90 con 0x7f276c06afd0 2023-11-24T06:09:08.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.950+0000 7f2772360700 1 -- 172.21.15.91:0/2345461054 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f276c06afd0 msgr2=0x7f276c0b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.950+0000 7f2772360700 1 --2- 172.21.15.91:0/2345461054 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f276c06afd0 0x7f276c0b59d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f27600098f0 tx=0x7f2760009c00 comp rx=0 tx=0).stop 2023-11-24T06:09:08.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.950+0000 7f2772360700 1 -- 172.21.15.91:0/2345461054 shutdown_connections 2023-11-24T06:09:08.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.950+0000 7f2772360700 1 --2- 172.21.15.91:0/2345461054 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f276c06afd0 0x7f276c0b59d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.950+0000 7f2772360700 1 --2- 172.21.15.91:0/2345461054 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f276c06b3b0 0x7f276c15e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.950+0000 7f2772360700 1 -- 172.21.15.91:0/2345461054 >> 172.21.15.91:0/2345461054 conn(0x7f276c0b2760 msgr2=0x7f276c0b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:08.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.950+0000 7f2772360700 1 -- 172.21.15.91:0/2345461054 shutdown_connections 2023-11-24T06:09:08.955 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.950+0000 7f2772360700 1 -- 172.21.15.91:0/2345461054 wait complete. 2023-11-24T06:09:08.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.951+0000 7f2772360700 1 Processor -- start 2023-11-24T06:09:08.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.952+0000 7f2772360700 1 -- start start 2023-11-24T06:09:08.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.952+0000 7f2772360700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f276c06afd0 0x7f276c0cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.952+0000 7f2772360700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f276c06b3b0 0x7f276c0ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.952+0000 7f2772360700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f276c072350 0x7f276c0aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.960 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.952+0000 7f2772360700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276c137540 con 0x7f276c072350 2023-11-24T06:09:08.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.952+0000 7f2772360700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276c134d20 con 0x7f276c06b3b0 2023-11-24T06:09:08.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.952+0000 7f2772360700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f276c0c6a00 con 0x7f276c06afd0 2023-11-24T06:09:08.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.952+0000 7f27708fd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f276c072350 0x7f276c0aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.952+0000 7f27708fd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f276c072350 0x7f276c0aedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59900/0 (socket says 172.21.15.91:59900) 2023-11-24T06:09:08.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.952+0000 7f27708fd700 1 -- 172.21.15.91:0/2344866788 learned_addr learned my addr 172.21.15.91:0/2344866788 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:08.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.953+0000 7f276bfff700 1 --2- 172.21.15.91:0/2344866788 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f276c06afd0 0x7f276c0cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.953+0000 7f276b7fe700 1 --2- 172.21.15.91:0/2344866788 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f276c06b3b0 0x7f276c0ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.961 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.953+0000 7f27708fd700 1 -- 172.21.15.91:0/2344866788 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f276c06afd0 msgr2=0x7f276c0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.953+0000 7f27708fd700 1 --2- 172.21.15.91:0/2344866788 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f276c06afd0 0x7f276c0cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.953+0000 7f27708fd700 1 -- 172.21.15.91:0/2344866788 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f276c06b3b0 msgr2=0x7f276c0ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:08.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.953+0000 7f27708fd700 1 --2- 172.21.15.91:0/2344866788 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f276c06b3b0 0x7f276c0ae8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:08.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.953+0000 7f27708fd700 1 -- 172.21.15.91:0/2344866788 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2760009550 con 0x7f276c072350 2023-11-24T06:09:08.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.953+0000 7f276b7fe700 1 --2- 172.21.15.91:0/2344866788 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f276c06b3b0 0x7f276c0ae8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:08.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.953+0000 7f276bfff700 1 --2- 172.21.15.91:0/2344866788 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f276c06afd0 0x7f276c0cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:08.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.953+0000 7f27708fd700 1 --2- 172.21.15.91:0/2344866788 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f276c072350 0x7f276c0aedf0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f276400cba0 tx=0x7f276400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:08.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.953+0000 7f2768ff9700 1 -- 172.21.15.91:0/2344866788 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276400ab80 con 0x7f276c072350 2023-11-24T06:09:08.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.953+0000 7f2768ff9700 1 -- 172.21.15.91:0/2344866788 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276400ad00 con 0x7f276c072350 2023-11-24T06:09:08.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.953+0000 7f2772360700 1 -- 172.21.15.91:0/2344866788 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f276c137260 con 0x7f276c072350 2023-11-24T06:09:08.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.953+0000 7f2768ff9700 1 -- 172.21.15.91:0/2344866788 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2764007310 con 0x7f276c072350 2023-11-24T06:09:08.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.953+0000 7f2772360700 1 -- 172.21.15.91:0/2344866788 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f276c134940 con 0x7f276c072350 2023-11-24T06:09:08.962 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.954+0000 7f2772360700 1 -- 172.21.15.91:0/2344866788 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f276c1344c0 con 0x7f276c072350 2023-11-24T06:09:08.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.954+0000 7f2768ff9700 1 -- 172.21.15.91:0/2344866788 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2764007470 con 0x7f276c072350 2023-11-24T06:09:08.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.955+0000 7f2768ff9700 1 --2- 172.21.15.91:0/2344866788 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f275806aed0 0x7f275806d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:08.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.955+0000 7f2768ff9700 1 -- 172.21.15.91:0/2344866788 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2764089380 con 0x7f276c072350 2023-11-24T06:09:08.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.955+0000 7f276bfff700 1 --2- 172.21.15.91:0/2344866788 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f275806aed0 0x7f275806d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:08.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.955+0000 7f276bfff700 1 --2- 172.21.15.91:0/2344866788 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f275806aed0 0x7f275806d390 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2760000c00 tx=0x7f2760008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:08.963 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:08.959+0000 7f2768ff9700 1 -- 172.21.15.91:0/2344866788 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f276c1344c0 con 0x7f276c072350 2023-11-24T06:09:09.185 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.182+0000 7f2772360700 1 -- 172.21.15.91:0/2344866788 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol4424", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f276c06b810 con 0x7f275806aed0 2023-11-24T06:09:09.188 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:09:09.186+0000 7fefd5c27700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T06:09:09.189 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:09:09.186+0000 7fefd5c27700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T06:09:09.189 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:09:09.186+0000 7fefd5c27700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T06:09:09.189 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:09:09.186+0000 7fefd5c27700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T06:09:09.189 INFO:tasks.ceph.mgr.y.smithi091.stderr:2023-11-24T06:09:09.186+0000 7fefd5c27700 -1 client.0 error registering admin socket command: (17) File exists 2023-11-24T06:09:09.215 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.212+0000 7f2768ff9700 1 -- 172.21.15.91:0/2344866788 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f276000bb70 con 0x7f275806aed0 2023-11-24T06:09:09.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.214+0000 7f2772360700 1 -- 172.21.15.91:0/2344866788 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f275806aed0 msgr2=0x7f275806d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:09.217 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.214+0000 7f2772360700 1 --2- 172.21.15.91:0/2344866788 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f275806aed0 0x7f275806d390 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2760000c00 tx=0x7f2760008030 comp rx=0 tx=0).stop 2023-11-24T06:09:09.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.214+0000 7f2772360700 1 -- 172.21.15.91:0/2344866788 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f276c072350 msgr2=0x7f276c0aedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:09.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.214+0000 7f2772360700 1 --2- 172.21.15.91:0/2344866788 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f276c072350 0x7f276c0aedf0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f276400cba0 tx=0x7f276400cf60 comp rx=0 tx=0).stop 2023-11-24T06:09:09.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.214+0000 7f2772360700 1 -- 172.21.15.91:0/2344866788 shutdown_connections 2023-11-24T06:09:09.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.214+0000 7f2772360700 1 --2- 172.21.15.91:0/2344866788 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f275806aed0 0x7f275806d390 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:09.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.214+0000 7f2772360700 1 --2- 172.21.15.91:0/2344866788 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f276c072350 0x7f276c0aedf0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:09.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.214+0000 7f2772360700 1 --2- 172.21.15.91:0/2344866788 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f276c06afd0 0x7f276c0cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:09.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.214+0000 7f2772360700 1 --2- 172.21.15.91:0/2344866788 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f276c06b3b0 0x7f276c0ae8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:09.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.214+0000 7f2772360700 1 -- 172.21.15.91:0/2344866788 >> 172.21.15.91:0/2344866788 conn(0x7f276c0b2760 msgr2=0x7f276c06e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:09.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.214+0000 7f2772360700 1 -- 172.21.15.91:0/2344866788 shutdown_connections 2023-11-24T06:09:09.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.214+0000 7f2772360700 1 -- 172.21.15.91:0/2344866788 wait complete. 2023-11-24T06:09:09.229 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol4424 2023-11-24T06:09:09.353 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.350+0000 7fc978a20700 1 Processor -- start 2023-11-24T06:09:09.353 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.350+0000 7fc978a20700 1 -- start start 2023-11-24T06:09:09.354 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.351+0000 7fc978a20700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9740cd5d0 0x7fc97406afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:09.355 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.351+0000 7fc978a20700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97406daf0 con 0x7fc9740cda50 2023-11-24T06:09:09.355 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.351+0000 7fc978a20700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9740cfe60 con 0x7fc9741459c0 2023-11-24T06:09:09.355 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.351+0000 7fc978a20700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9740d0140 con 0x7fc9740cd5d0 2023-11-24T06:09:09.355 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.351+0000 7fc972d9d700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fc9740cda50 0x7fc97406e260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54858/0 (socket says 172.21.15.91:54858) 2023-11-24T06:09:09.355 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.351+0000 7fc972d9d700 1 -- 172.21.15.91:0/2549824219 learned_addr learned my addr 172.21.15.91:0/2549824219 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:09.355 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.351+0000 7fc971d9b700 1 --2- 172.21.15.91:0/2549824219 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9740cd5d0 0x7fc97406afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:09.355 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.351+0000 7fc97159a700 1 -- 172.21.15.91:0/2549824219 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101363446 0 0) 0x7fc97406daf0 con 0x7fc9740cda50 2023-11-24T06:09:09.355 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.351+0000 7fc97159a700 1 -- 172.21.15.91:0/2549824219 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc95c0035b0 con 0x7fc9740cda50 2023-11-24T06:09:09.355 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.351+0000 7fc97159a700 1 -- 172.21.15.91:0/2549824219 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186154938 0 0) 0x7fc9740cfe60 con 0x7fc9741459c0 2023-11-24T06:09:09.355 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.351+0000 7fc97159a700 1 -- 172.21.15.91:0/2549824219 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc97406daf0 con 0x7fc9741459c0 2023-11-24T06:09:09.355 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.351+0000 7fc971d9b700 1 -- 172.21.15.91:0/2549824219 >> v1:172.21.15.94:6789/0 conn(0x7fc9741459c0 legacy=0x7fc97415c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:09.355 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.351+0000 7fc971d9b700 1 -- 172.21.15.91:0/2549824219 >> v1:172.21.15.91:6789/0 conn(0x7fc9740cda50 legacy=0x7fc97406e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:09.356 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.351+0000 7fc971d9b700 1 -- 172.21.15.91:0/2549824219 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9740d0420 con 0x7fc9740cd5d0 2023-11-24T06:09:09.356 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.352+0000 7fc971d9b700 1 --2- 172.21.15.91:0/2549824219 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9740cd5d0 0x7fc97406afd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc964009870 tx=0x7fc964009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f422fa5bf83d3f27 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:09.356 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.352+0000 7fc97159a700 1 -- 172.21.15.91:0/2549824219 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc964011830 con 0x7fc9740cd5d0 2023-11-24T06:09:09.356 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.352+0000 7fc97159a700 1 -- 172.21.15.91:0/2549824219 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc964011990 con 0x7fc9740cd5d0 2023-11-24T06:09:09.356 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.352+0000 7fc97159a700 1 -- 172.21.15.91:0/2549824219 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc964011d20 con 0x7fc9740cd5d0 2023-11-24T06:09:09.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.352+0000 7fc978a20700 1 -- 172.21.15.91:0/2549824219 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9740cd5d0 msgr2=0x7fc97406afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:09.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.352+0000 7fc978a20700 1 --2- 172.21.15.91:0/2549824219 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9740cd5d0 0x7fc97406afd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc964009870 tx=0x7fc964009b80 comp rx=0 tx=0).stop 2023-11-24T06:09:09.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.352+0000 7fc978a20700 1 -- 172.21.15.91:0/2549824219 shutdown_connections 2023-11-24T06:09:09.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.352+0000 7fc978a20700 1 --2- 172.21.15.91:0/2549824219 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9740cd5d0 0x7fc97406afd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:09.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.352+0000 7fc978a20700 1 -- 172.21.15.91:0/2549824219 >> 172.21.15.91:0/2549824219 conn(0x7fc9740b27b0 msgr2=0x7fc9740b2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:09.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.352+0000 7fc978a20700 1 -- 172.21.15.91:0/2549824219 shutdown_connections 2023-11-24T06:09:09.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.353+0000 7fc978a20700 1 -- 172.21.15.91:0/2549824219 wait complete. 2023-11-24T06:09:09.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.354+0000 7fc978a20700 1 Processor -- start 2023-11-24T06:09:09.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.354+0000 7fc978a20700 1 -- start start 2023-11-24T06:09:09.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.354+0000 7fc978a20700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9740cd5d0 0x7fc974144230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:09.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.354+0000 7fc978a20700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9740cda50 0x7fc9740cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:09.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc978a20700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc9741459c0 0x7fc9740cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:09.360 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc978a20700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97413fb50 con 0x7fc9741459c0 2023-11-24T06:09:09.360 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc978a20700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9741333b0 con 0x7fc9740cda50 2023-11-24T06:09:09.360 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc978a20700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9741307e0 con 0x7fc9740cd5d0 2023-11-24T06:09:09.360 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc972d9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc9741459c0 0x7fc9740cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:09.360 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc972d9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc9741459c0 0x7fc9740cc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59930/0 (socket says 172.21.15.91:59930) 2023-11-24T06:09:09.360 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc972d9d700 1 -- 172.21.15.91:0/2585815408 learned_addr learned my addr 172.21.15.91:0/2585815408 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:09.360 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc972d9d700 1 -- 172.21.15.91:0/2585815408 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9740cd5d0 msgr2=0x7fc974144230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:09.360 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc97259c700 1 --2- 172.21.15.91:0/2585815408 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9740cd5d0 0x7fc974144230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:09.361 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc971d9b700 1 --2- 172.21.15.91:0/2585815408 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9740cda50 0x7fc9740cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:09.361 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc972d9d700 1 --2- 172.21.15.91:0/2585815408 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9740cd5d0 0x7fc974144230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:09.361 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc972d9d700 1 -- 172.21.15.91:0/2585815408 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9740cda50 msgr2=0x7fc9740cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:09.361 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc972d9d700 1 --2- 172.21.15.91:0/2585815408 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9740cda50 0x7fc9740cbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:09.361 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc972d9d700 1 -- 172.21.15.91:0/2585815408 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9640094a0 con 0x7fc9741459c0 2023-11-24T06:09:09.361 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc97259c700 1 --2- 172.21.15.91:0/2585815408 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9740cd5d0 0x7fc974144230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:09.361 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc971d9b700 1 --2- 172.21.15.91:0/2585815408 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9740cda50 0x7fc9740cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:09.361 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc972d9d700 1 --2- 172.21.15.91:0/2585815408 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc9741459c0 0x7fc9740cc130 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc96c00cbc0 tx=0x7fc96c00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:09.361 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc962ffd700 1 -- 172.21.15.91:0/2585815408 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c00aaf0 con 0x7fc9741459c0 2023-11-24T06:09:09.361 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc962ffd700 1 -- 172.21.15.91:0/2585815408 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc96c00ac50 con 0x7fc9741459c0 2023-11-24T06:09:09.361 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc962ffd700 1 -- 172.21.15.91:0/2585815408 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c012400 con 0x7fc9741459c0 2023-11-24T06:09:09.361 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc978a20700 1 -- 172.21.15.91:0/2585815408 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc974130ac0 con 0x7fc9741459c0 2023-11-24T06:09:09.362 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.355+0000 7fc978a20700 1 -- 172.21.15.91:0/2585815408 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc974133690 con 0x7fc9741459c0 2023-11-24T06:09:09.362 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.356+0000 7fc978a20700 1 -- 172.21.15.91:0/2585815408 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc974131110 con 0x7fc9741459c0 2023-11-24T06:09:09.362 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.357+0000 7fc962ffd700 1 -- 172.21.15.91:0/2585815408 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc96c01b3b0 con 0x7fc9741459c0 2023-11-24T06:09:09.362 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.357+0000 7fc962ffd700 1 --2- 172.21.15.91:0/2585815408 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc95406aed0 0x7fc95406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:09.362 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.357+0000 7fc962ffd700 1 -- 172.21.15.91:0/2585815408 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc96c0891c0 con 0x7fc9741459c0 2023-11-24T06:09:09.363 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.357+0000 7fc97259c700 1 --2- 172.21.15.91:0/2585815408 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc95406aed0 0x7fc95406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:09.363 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.357+0000 7fc97259c700 1 --2- 172.21.15.91:0/2585815408 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc95406aed0 0x7fc95406d390 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc9680011e0 tx=0x7fc968009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:09.365 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.361+0000 7fc962ffd700 1 -- 172.21.15.91:0/2585815408 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc96c0543b0 con 0x7fc9741459c0 2023-11-24T06:09:09.592 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.589+0000 7fc978a20700 1 -- 172.21.15.91:0/2585815408 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol4424", "target": ["mon-mgr", ""]}) v1 -- 0x7fc97406b510 con 0x7fc95406aed0 2023-11-24T06:09:13.046 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.595+0000 7fc962ffd700 1 -- 172.21.15.91:0/2585815408 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fc9680013d0 con 0x7fc95406aed0 2023-11-24T06:09:13.046 INFO:teuthology.orchestra.run.smithi091.stdout:/volumes/_nogroup/subvol4424/c99cae04-449b-4c27-b3ea-7054d14b2916 2023-11-24T06:09:13.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.597+0000 7fc978a20700 1 -- 172.21.15.91:0/2585815408 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc95406aed0 msgr2=0x7fc95406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:13.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.597+0000 7fc978a20700 1 --2- 172.21.15.91:0/2585815408 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc95406aed0 0x7fc95406d390 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc9680011e0 tx=0x7fc968009040 comp rx=0 tx=0).stop 2023-11-24T06:09:13.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.597+0000 7fc978a20700 1 -- 172.21.15.91:0/2585815408 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc9741459c0 msgr2=0x7fc9740cc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:13.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.597+0000 7fc978a20700 1 --2- 172.21.15.91:0/2585815408 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc9741459c0 0x7fc9740cc130 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc96c00cbc0 tx=0x7fc96c00cf80 comp rx=0 tx=0).stop 2023-11-24T06:09:13.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.597+0000 7fc978a20700 1 -- 172.21.15.91:0/2585815408 shutdown_connections 2023-11-24T06:09:13.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.597+0000 7fc978a20700 1 --2- 172.21.15.91:0/2585815408 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc95406aed0 0x7fc95406d390 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.597+0000 7fc978a20700 1 --2- 172.21.15.91:0/2585815408 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc9741459c0 0x7fc9740cc130 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.597+0000 7fc978a20700 1 --2- 172.21.15.91:0/2585815408 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc9740cd5d0 0x7fc974144230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.597+0000 7fc978a20700 1 --2- 172.21.15.91:0/2585815408 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc9740cda50 0x7fc9740cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.597+0000 7fc978a20700 1 -- 172.21.15.91:0/2585815408 >> 172.21.15.91:0/2585815408 conn(0x7fc9740b27b0 msgr2=0x7fc97415eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:13.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.597+0000 7fc978a20700 1 -- 172.21.15.91:0/2585815408 shutdown_connections 2023-11-24T06:09:13.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:09.597+0000 7fc978a20700 1 -- 172.21.15.91:0/2585815408 wait complete. 2023-11-24T06:09:13.048 DEBUG:tasks.cephfs.test_volumes:filling subvolume subvol4424 with 200 files each 1MB size under directory volumes/_nogroup/subvol4424/c99cae04-449b-4c27-b3ea-7054d14b2916 2023-11-24T06:09:13.049 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-11-24T06:09:13.049 DEBUG:teuthology.orchestra.run.smithi091:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T06:09:13.097 DEBUG:teuthology.orchestra.run.smithi094:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-11-24T06:09:13.139 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-11-24T06:09:13.139 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:09:13.140 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:09:13.160 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:09:13.160 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:09:13.220 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:09:13.390 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.387+0000 7f3d6dae7700 1 Processor -- start 2023-11-24T06:09:13.390 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.387+0000 7f3d6dae7700 1 -- start start 2023-11-24T06:09:13.392 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.388+0000 7f3d6dae7700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3d680cfd00 0x7f3d680b4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:13.392 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.388+0000 7f3d6dae7700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3d680d00e0 0x7f3d680d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:13.392 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.388+0000 7f3d6dae7700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d680c5be0 con 0x7f3d680ce690 2023-11-24T06:09:13.392 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.388+0000 7f3d6dae7700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d680cdad0 con 0x7f3d680cfd00 2023-11-24T06:09:13.392 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.388+0000 7f3d6dae7700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d680cd230 con 0x7f3d680d00e0 2023-11-24T06:09:13.392 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.388+0000 7f3d677fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3d680cfd00 0x7f3d680b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:13.393 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.388+0000 7f3d66ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3d680d00e0 0x7f3d680d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:13.393 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.388+0000 7f3d67fff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7f3d680ce690 0x7f3d680ceb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.91:56664/0 (socket says 172.21.15.91:56664) 2023-11-24T06:09:13.393 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.389+0000 7f3d66ffd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3d680d00e0 0x7f3d680d0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:43368/0 (socket says 172.21.15.91:43368) 2023-11-24T06:09:13.393 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.389+0000 7f3d67fff700 1 -- 172.21.15.91:0/3509936959 learned_addr learned my addr 172.21.15.91:0/3509936959 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:13.393 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.389+0000 7f3d677fe700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3d680cfd00 0x7f3d680b4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:44326/0 (socket says 172.21.15.91:44326) 2023-11-24T06:09:13.393 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.389+0000 7f3d677fe700 1 -- 172.21.15.91:0/3509936959 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3d680d00e0 msgr2=0x7f3d680d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:13.393 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.389+0000 7f3d677fe700 1 --2- 172.21.15.91:0/3509936959 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3d680d00e0 0x7f3d680d0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.393 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.389+0000 7f3d677fe700 1 -- 172.21.15.91:0/3509936959 >> v1:172.21.15.94:6789/0 conn(0x7f3d680ce690 legacy=0x7f3d680ceb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:13.393 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.389+0000 7f3d677fe700 1 -- 172.21.15.91:0/3509936959 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d680cd510 con 0x7f3d680cfd00 2023-11-24T06:09:13.393 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.389+0000 7f3d667fc700 1 -- 172.21.15.91:0/3509936959 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1903382053 0 0) 0x7f3d680c5be0 con 0x7f3d680ce690 2023-11-24T06:09:13.393 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.389+0000 7f3d677fe700 1 --2- 172.21.15.91:0/3509936959 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d680cfd00 0x7f3d680b4390 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009840 tx=0x7f3d50009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff409782d637acd4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:13.393 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.389+0000 7f3d667fc700 1 -- 172.21.15.91:0/3509936959 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d50011830 con 0x7f3d680cfd00 2023-11-24T06:09:13.394 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.389+0000 7f3d667fc700 1 -- 172.21.15.91:0/3509936959 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d50011990 con 0x7f3d680cfd00 2023-11-24T06:09:13.394 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.389+0000 7f3d667fc700 1 -- 172.21.15.91:0/3509936959 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d50011d20 con 0x7f3d680cfd00 2023-11-24T06:09:13.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.390+0000 7f3d6dae7700 1 -- 172.21.15.91:0/3509936959 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d680cfd00 msgr2=0x7f3d680b4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:13.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.390+0000 7f3d6dae7700 1 --2- 172.21.15.91:0/3509936959 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d680cfd00 0x7f3d680b4390 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009840 tx=0x7f3d50009b50 comp rx=0 tx=0).stop 2023-11-24T06:09:13.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.390+0000 7f3d6dae7700 1 -- 172.21.15.91:0/3509936959 shutdown_connections 2023-11-24T06:09:13.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.390+0000 7f3d6dae7700 1 --2- 172.21.15.91:0/3509936959 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d680cfd00 0x7f3d680b4390 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.390+0000 7f3d6dae7700 1 --2- 172.21.15.91:0/3509936959 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3d680d00e0 0x7f3d680d0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.390+0000 7f3d6dae7700 1 -- 172.21.15.91:0/3509936959 >> 172.21.15.91:0/3509936959 conn(0x7f3d680b3870 msgr2=0x7f3d680b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:13.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.390+0000 7f3d6dae7700 1 -- 172.21.15.91:0/3509936959 shutdown_connections 2023-11-24T06:09:13.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.390+0000 7f3d6dae7700 1 -- 172.21.15.91:0/3509936959 wait complete. 2023-11-24T06:09:13.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.391+0000 7f3d6dae7700 1 Processor -- start 2023-11-24T06:09:13.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.392+0000 7f3d6dae7700 1 -- start start 2023-11-24T06:09:13.397 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.392+0000 7f3d6dae7700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3d680ce690 0x7f3d681f2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:13.397 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.392+0000 7f3d6dae7700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d680cfd00 0x7f3d68159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:13.397 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d6dae7700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3d680d00e0 0x7f3d680ca950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:13.397 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d6dae7700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6815d1a0 con 0x7f3d680d00e0 2023-11-24T06:09:13.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d6dae7700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6815c560 con 0x7f3d680cfd00 2023-11-24T06:09:13.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d6dae7700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6815c800 con 0x7f3d680ce690 2023-11-24T06:09:13.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d67fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3d680d00e0 0x7f3d680ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:13.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d67fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3d680d00e0 0x7f3d680ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59944/0 (socket says 172.21.15.91:59944) 2023-11-24T06:09:13.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d67fff700 1 -- 172.21.15.91:0/3125383916 learned_addr learned my addr 172.21.15.91:0/3125383916 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:13.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d66ffd700 1 --2- 172.21.15.91:0/3125383916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d680cfd00 0x7f3d68159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:13.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d677fe700 1 --2- 172.21.15.91:0/3125383916 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3d680ce690 0x7f3d681f2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:13.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d67fff700 1 -- 172.21.15.91:0/3125383916 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3d680ce690 msgr2=0x7f3d681f2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:13.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d67fff700 1 --2- 172.21.15.91:0/3125383916 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3d680ce690 0x7f3d681f2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d67fff700 1 -- 172.21.15.91:0/3125383916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d680cfd00 msgr2=0x7f3d68159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:13.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d67fff700 1 --2- 172.21.15.91:0/3125383916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d680cfd00 0x7f3d68159310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d67fff700 1 -- 172.21.15.91:0/3125383916 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d500094a0 con 0x7f3d680d00e0 2023-11-24T06:09:13.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d67fff700 1 --2- 172.21.15.91:0/3125383916 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3d680d00e0 0x7f3d680ca950 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000cca0 tx=0x7f3d6000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:13.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d5ffff700 1 -- 172.21.15.91:0/3125383916 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6000ab70 con 0x7f3d680d00e0 2023-11-24T06:09:13.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d5ffff700 1 -- 172.21.15.91:0/3125383916 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6000acd0 con 0x7f3d680d00e0 2023-11-24T06:09:13.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d6dae7700 1 -- 172.21.15.91:0/3125383916 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d6815cae0 con 0x7f3d680d00e0 2023-11-24T06:09:13.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d5ffff700 1 -- 172.21.15.91:0/3125383916 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d60007310 con 0x7f3d680d00e0 2023-11-24T06:09:13.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.393+0000 7f3d6dae7700 1 -- 172.21.15.91:0/3125383916 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d68159850 con 0x7f3d680d00e0 2023-11-24T06:09:13.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.394+0000 7f3d6dae7700 1 -- 172.21.15.91:0/3125383916 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d6815d2e0 con 0x7f3d680d00e0 2023-11-24T06:09:13.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.395+0000 7f3d5ffff700 1 -- 172.21.15.91:0/3125383916 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f3d60007470 con 0x7f3d680d00e0 2023-11-24T06:09:13.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.395+0000 7f3d5ffff700 1 --2- 172.21.15.91:0/3125383916 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3d4806af60 0x7f3d4806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:13.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.395+0000 7f3d677fe700 1 --2- 172.21.15.91:0/3125383916 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3d4806af60 0x7f3d4806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:13.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.395+0000 7f3d5ffff700 1 -- 172.21.15.91:0/3125383916 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d600893a0 con 0x7f3d680d00e0 2023-11-24T06:09:13.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.396+0000 7f3d677fe700 1 --2- 172.21.15.91:0/3125383916 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3d4806af60 0x7f3d4806d420 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009fd0 tx=0x7f3d50008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:13.403 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.400+0000 7f3d5ffff700 1 -- 172.21.15.91:0/3125383916 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d6815d2e0 con 0x7f3d680d00e0 2023-11-24T06:09:13.594 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.592+0000 7f3d6dae7700 1 -- 172.21.15.91:0/3125383916 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d6815d2e0 con 0x7f3d680d00e0 2023-11-24T06:09:13.596 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.593+0000 7f3d5ffff700 1 -- 172.21.15.91:0/3125383916 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16485 (secure 0 0 0) 0x7f3d6815d2e0 con 0x7f3d680d00e0 2023-11-24T06:09:13.596 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:13.596 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":34,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:09:00.042725+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:08:56.059841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:08:57.408988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:09:13.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.595+0000 7f3d6dae7700 1 -- 172.21.15.91:0/3125383916 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3d4806af60 msgr2=0x7f3d4806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:13.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.595+0000 7f3d6dae7700 1 --2- 172.21.15.91:0/3125383916 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3d4806af60 0x7f3d4806d420 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009fd0 tx=0x7f3d50008030 comp rx=0 tx=0).stop 2023-11-24T06:09:13.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.595+0000 7f3d6dae7700 1 -- 172.21.15.91:0/3125383916 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3d680d00e0 msgr2=0x7f3d680ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:13.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.595+0000 7f3d6dae7700 1 --2- 172.21.15.91:0/3125383916 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3d680d00e0 0x7f3d680ca950 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000cca0 tx=0x7f3d6000a060 comp rx=0 tx=0).stop 2023-11-24T06:09:13.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.595+0000 7f3d6dae7700 1 -- 172.21.15.91:0/3125383916 shutdown_connections 2023-11-24T06:09:13.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.595+0000 7f3d6dae7700 1 --2- 172.21.15.91:0/3125383916 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3d4806af60 0x7f3d4806d420 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.595+0000 7f3d6dae7700 1 --2- 172.21.15.91:0/3125383916 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3d680d00e0 0x7f3d680ca950 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.595+0000 7f3d6dae7700 1 --2- 172.21.15.91:0/3125383916 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3d680ce690 0x7f3d681f2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.595+0000 7f3d6dae7700 1 --2- 172.21.15.91:0/3125383916 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3d680cfd00 0x7f3d68159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.595+0000 7f3d6dae7700 1 -- 172.21.15.91:0/3125383916 >> 172.21.15.91:0/3125383916 conn(0x7f3d680b3870 msgr2=0x7f3d680b6970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:13.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.595+0000 7f3d6dae7700 1 -- 172.21.15.91:0/3125383916 shutdown_connections 2023-11-24T06:09:13.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.595+0000 7f3d6dae7700 1 -- 172.21.15.91:0/3125383916 wait complete. 2023-11-24T06:09:13.610 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:08:56.059841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:08:57.408988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:09:13.611 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:09:13.734 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.731+0000 7f7e9ff68700 1 Processor -- start 2023-11-24T06:09:13.734 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.731+0000 7f7e9ff68700 1 -- start start 2023-11-24T06:09:13.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9ff68700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7e980722c0 0x7f7e980b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:13.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9ff68700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f7e981440f0 0x7f7e9806e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:13.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9ff68700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e98143ec0 con 0x7f7e98143ae0 2023-11-24T06:09:13.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9ff68700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e980b40a0 con 0x7f7e981440f0 2023-11-24T06:09:13.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9ff68700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e980b3ae0 con 0x7f7e980722c0 2023-11-24T06:09:13.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9e505700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f7e981440f0 0x7f7e9806e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:13.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9dd04700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f7e98143ae0 0x7f7e98071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54882/0 (socket says 172.21.15.91:54882) 2023-11-24T06:09:13.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9dd04700 1 -- 172.21.15.91:0/1833015926 learned_addr learned my addr 172.21.15.91:0/1833015926 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:13.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9d503700 1 --2- 172.21.15.91:0/1833015926 >> v2:172.21.15.94:3300/0 conn(0x7f7e980722c0 0x7f7e980b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:13.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9e505700 1 -- 172.21.15.91:0/1833015926 >> v2:172.21.15.94:3300/0 conn(0x7f7e980722c0 msgr2=0x7f7e980b51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:13.744 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9e505700 1 --2- 172.21.15.91:0/1833015926 >> v2:172.21.15.94:3300/0 conn(0x7f7e980722c0 0x7f7e980b51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9e505700 1 -- 172.21.15.91:0/1833015926 >> v1:172.21.15.91:6789/0 conn(0x7f7e98143ae0 legacy=0x7f7e98071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:13.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9e505700 1 -- 172.21.15.91:0/1833015926 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e980b3dc0 con 0x7f7e981440f0 2023-11-24T06:09:13.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9d503700 1 --2- 172.21.15.91:0/1833015926 >> v2:172.21.15.94:3300/0 conn(0x7f7e980722c0 0x7f7e980b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:13.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9e505700 1 --2- 172.21.15.91:0/1833015926 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7e981440f0 0x7f7e9806e510 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7e980d1b00 tx=0x7f7e9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f4d08d8b7cd648d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:13.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9cd02700 1 -- 172.21.15.91:0/1833015926 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9000e070 con 0x7f7e981440f0 2023-11-24T06:09:13.745 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9cd02700 1 -- 172.21.15.91:0/1833015926 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9000bbd0 con 0x7f7e981440f0 2023-11-24T06:09:13.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.732+0000 7f7e9cd02700 1 -- 172.21.15.91:0/1833015926 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e90012430 con 0x7f7e981440f0 2023-11-24T06:09:13.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.733+0000 7f7e9ff68700 1 -- 172.21.15.91:0/1833015926 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7e981440f0 msgr2=0x7f7e9806e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:13.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.733+0000 7f7e9ff68700 1 --2- 172.21.15.91:0/1833015926 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7e981440f0 0x7f7e9806e510 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f7e980d1b00 tx=0x7f7e9000b330 comp rx=0 tx=0).stop 2023-11-24T06:09:13.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.733+0000 7f7e9ff68700 1 -- 172.21.15.91:0/1833015926 shutdown_connections 2023-11-24T06:09:13.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.733+0000 7f7e9ff68700 1 --2- 172.21.15.91:0/1833015926 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7e981440f0 0x7f7e9806e510 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.746 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.733+0000 7f7e9ff68700 1 --2- 172.21.15.91:0/1833015926 >> v2:172.21.15.94:3300/0 conn(0x7f7e980722c0 0x7f7e980b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.733+0000 7f7e9ff68700 1 -- 172.21.15.91:0/1833015926 >> 172.21.15.91:0/1833015926 conn(0x7f7e980b2740 msgr2=0x7f7e980b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:13.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.733+0000 7f7e9ff68700 1 -- 172.21.15.91:0/1833015926 shutdown_connections 2023-11-24T06:09:13.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.733+0000 7f7e9ff68700 1 -- 172.21.15.91:0/1833015926 wait complete. 2023-11-24T06:09:13.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.734+0000 7f7e9ff68700 1 Processor -- start 2023-11-24T06:09:13.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.734+0000 7f7e9ff68700 1 -- start start 2023-11-24T06:09:13.747 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9ff68700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e980722c0 0x7f7e98158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:13.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9ff68700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7e98143ae0 0x7f7e980c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:13.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9ff68700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e981440f0 0x7f7e980c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:13.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9ff68700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e981593d0 con 0x7f7e98143ae0 2023-11-24T06:09:13.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9ff68700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9814f100 con 0x7f7e980722c0 2023-11-24T06:09:13.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9ff68700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9814c350 con 0x7f7e981440f0 2023-11-24T06:09:13.748 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9d503700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7e98143ae0 0x7f7e980c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:13.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9d503700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7e98143ae0 0x7f7e980c59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59954/0 (socket says 172.21.15.91:59954) 2023-11-24T06:09:13.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9d503700 1 -- 172.21.15.91:0/1761420315 learned_addr learned my addr 172.21.15.91:0/1761420315 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:13.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9dd04700 1 --2- 172.21.15.91:0/1761420315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e980722c0 0x7f7e98158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:13.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9e505700 1 --2- 172.21.15.91:0/1761420315 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e981440f0 0x7f7e980c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:13.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9d503700 1 -- 172.21.15.91:0/1761420315 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e981440f0 msgr2=0x7f7e980c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:13.749 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9d503700 1 --2- 172.21.15.91:0/1761420315 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e981440f0 0x7f7e980c5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9d503700 1 -- 172.21.15.91:0/1761420315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e980722c0 msgr2=0x7f7e98158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:13.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9d503700 1 --2- 172.21.15.91:0/1761420315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e980722c0 0x7f7e98158e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9d503700 1 -- 172.21.15.91:0/1761420315 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e90009a70 con 0x7f7e98143ae0 2023-11-24T06:09:13.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.735+0000 7f7e9e505700 1 --2- 172.21.15.91:0/1761420315 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e981440f0 0x7f7e980c5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:13.750 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.736+0000 7f7e9d503700 1 --2- 172.21.15.91:0/1761420315 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7e98143ae0 0x7f7e980c59b0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7e9400c750 tx=0x7f7e9400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:13.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.736+0000 7f7e8e7fc700 1 -- 172.21.15.91:0/1761420315 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e94011800 con 0x7f7e98143ae0 2023-11-24T06:09:13.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.736+0000 7f7e8e7fc700 1 -- 172.21.15.91:0/1761420315 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e94011960 con 0x7f7e98143ae0 2023-11-24T06:09:13.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.736+0000 7f7e9ff68700 1 -- 172.21.15.91:0/1761420315 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9814c610 con 0x7f7e98143ae0 2023-11-24T06:09:13.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.736+0000 7f7e8e7fc700 1 -- 172.21.15.91:0/1761420315 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e94011cd0 con 0x7f7e98143ae0 2023-11-24T06:09:13.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.736+0000 7f7e9ff68700 1 -- 172.21.15.91:0/1761420315 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e9814cbd0 con 0x7f7e98143ae0 2023-11-24T06:09:13.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.737+0000 7f7e9ff68700 1 -- 172.21.15.91:0/1761420315 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e98075480 con 0x7f7e98143ae0 2023-11-24T06:09:13.751 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.737+0000 7f7e8e7fc700 1 -- 172.21.15.91:0/1761420315 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f7e94021350 con 0x7f7e98143ae0 2023-11-24T06:09:13.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.737+0000 7f7e8e7fc700 1 --2- 172.21.15.91:0/1761420315 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7e7c06af60 0x7f7e7c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:13.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.737+0000 7f7e8e7fc700 1 -- 172.21.15.91:0/1761420315 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e94089720 con 0x7f7e98143ae0 2023-11-24T06:09:13.752 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.737+0000 7f7e9dd04700 1 --2- 172.21.15.91:0/1761420315 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7e7c06af60 0x7f7e7c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:13.753 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.738+0000 7f7e9dd04700 1 --2- 172.21.15.91:0/1761420315 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7e7c06af60 0x7f7e7c06d420 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7e88005910 tx=0x7f7e88009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:13.753 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.742+0000 7f7e8e7fc700 1 -- 172.21.15.91:0/1761420315 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e940545d0 con 0x7f7e98143ae0 2023-11-24T06:09:13.936 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.933+0000 7f7e9ff68700 1 -- 172.21.15.91:0/1761420315 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e980b5d20 con 0x7f7e98143ae0 2023-11-24T06:09:13.936 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.933+0000 7f7e8e7fc700 1 -- 172.21.15.91:0/1761420315 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7e94057bf0 con 0x7f7e98143ae0 2023-11-24T06:09:13.937 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:09:13.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.935+0000 7f7e9ff68700 1 -- 172.21.15.91:0/1761420315 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7e7c06af60 msgr2=0x7f7e7c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:13.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.935+0000 7f7e9ff68700 1 --2- 172.21.15.91:0/1761420315 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7e7c06af60 0x7f7e7c06d420 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7e88005910 tx=0x7f7e88009040 comp rx=0 tx=0).stop 2023-11-24T06:09:13.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.935+0000 7f7e9ff68700 1 -- 172.21.15.91:0/1761420315 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7e98143ae0 msgr2=0x7f7e980c59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:13.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.935+0000 7f7e9ff68700 1 --2- 172.21.15.91:0/1761420315 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7e98143ae0 0x7f7e980c59b0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7e9400c750 tx=0x7f7e9400ca60 comp rx=0 tx=0).stop 2023-11-24T06:09:13.939 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.935+0000 7f7e9ff68700 1 -- 172.21.15.91:0/1761420315 shutdown_connections 2023-11-24T06:09:13.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.935+0000 7f7e9ff68700 1 --2- 172.21.15.91:0/1761420315 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7e7c06af60 0x7f7e7c06d420 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.935+0000 7f7e9ff68700 1 --2- 172.21.15.91:0/1761420315 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7e980722c0 0x7f7e98158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.935+0000 7f7e9ff68700 1 --2- 172.21.15.91:0/1761420315 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7e981440f0 0x7f7e980c5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.935+0000 7f7e9ff68700 1 --2- 172.21.15.91:0/1761420315 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7e98143ae0 0x7f7e980c59b0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:13.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.935+0000 7f7e9ff68700 1 -- 172.21.15.91:0/1761420315 >> 172.21.15.91:0/1761420315 conn(0x7f7e980b2740 msgr2=0x7f7e980b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:13.940 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.935+0000 7f7e9ff68700 1 -- 172.21.15.91:0/1761420315 shutdown_connections 2023-11-24T06:09:13.941 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:13.935+0000 7f7e9ff68700 1 -- 172.21.15.91:0/1761420315 wait complete. 2023-11-24T06:09:13.950 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:09:14.074 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.071+0000 7f6df60fa700 1 Processor -- start 2023-11-24T06:09:14.074 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.071+0000 7f6df60fa700 1 -- start start 2023-11-24T06:09:14.076 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.072+0000 7f6df60fa700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6df006b620 0x7f6df00b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.072+0000 7f6df60fa700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df01629c0 con 0x7f6df00b5ab0 2023-11-24T06:09:14.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.072+0000 7f6df60fa700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df0071f50 con 0x7f6df006ee60 2023-11-24T06:09:14.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.072+0000 7f6df60fa700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df0072230 con 0x7f6df006b620 2023-11-24T06:09:14.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.072+0000 7f6de6ffd700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f6df00b5ab0 0x7f6df0161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54896/0 (socket says 172.21.15.91:54896) 2023-11-24T06:09:14.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.072+0000 7f6de6ffd700 1 -- 172.21.15.91:0/2926510876 learned_addr learned my addr 172.21.15.91:0/2926510876 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:14.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.072+0000 7f6def7fe700 1 --2- 172.21.15.91:0/2926510876 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6df006b620 0x7f6df00b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:14.077 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.072+0000 7f6deeffd700 1 -- 172.21.15.91:0/2926510876 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205475716 0 0) 0x7f6df01629c0 con 0x7f6df00b5ab0 2023-11-24T06:09:14.078 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.072+0000 7f6deeffd700 1 -- 172.21.15.91:0/2926510876 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ddc0035b0 con 0x7f6df00b5ab0 2023-11-24T06:09:14.078 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.072+0000 7f6deeffd700 1 -- 172.21.15.91:0/2926510876 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277888904 0 0) 0x7f6df0071f50 con 0x7f6df006ee60 2023-11-24T06:09:14.078 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.072+0000 7f6deeffd700 1 -- 172.21.15.91:0/2926510876 --> v1:172.21.15.94:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df01629c0 con 0x7f6df006ee60 2023-11-24T06:09:14.078 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.073+0000 7f6deeffd700 1 -- 172.21.15.91:0/2926510876 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2748809498 0 0) 0x7f6ddc0035b0 con 0x7f6df00b5ab0 2023-11-24T06:09:14.078 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.073+0000 7f6def7fe700 1 -- 172.21.15.91:0/2926510876 >> v1:172.21.15.94:6789/0 conn(0x7f6df006ee60 legacy=0x7f6df01622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.078 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.073+0000 7f6def7fe700 1 -- 172.21.15.91:0/2926510876 >> v1:172.21.15.91:6789/0 conn(0x7f6df00b5ab0 legacy=0x7f6df0161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.079 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.073+0000 7f6def7fe700 1 -- 172.21.15.91:0/2926510876 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df0072510 con 0x7f6df006b620 2023-11-24T06:09:14.079 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.073+0000 7f6def7fe700 1 --2- 172.21.15.91:0/2926510876 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6df006b620 0x7f6df00b3a40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009870 tx=0x7f6de0009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a60f5cc55cec5778 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:14.079 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.073+0000 7f6deeffd700 1 -- 172.21.15.91:0/2926510876 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd8001670 con 0x7f6df006b620 2023-11-24T06:09:14.079 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.073+0000 7f6deeffd700 1 -- 172.21.15.91:0/2926510876 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6de0004030 con 0x7f6df006b620 2023-11-24T06:09:14.079 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.073+0000 7f6deeffd700 1 -- 172.21.15.91:0/2926510876 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de00043a0 con 0x7f6df006b620 2023-11-24T06:09:14.079 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.073+0000 7f6df60fa700 1 -- 172.21.15.91:0/2926510876 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6df006b620 msgr2=0x7f6df00b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.080 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.074+0000 7f6df60fa700 1 --2- 172.21.15.91:0/2926510876 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6df006b620 0x7f6df00b3a40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009870 tx=0x7f6de0009b80 comp rx=0 tx=0).stop 2023-11-24T06:09:14.080 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.074+0000 7f6df60fa700 1 -- 172.21.15.91:0/2926510876 shutdown_connections 2023-11-24T06:09:14.080 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.074+0000 7f6df60fa700 1 --2- 172.21.15.91:0/2926510876 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6df006b620 0x7f6df00b3a40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.080 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.074+0000 7f6df60fa700 1 -- 172.21.15.91:0/2926510876 >> 172.21.15.91:0/2926510876 conn(0x7f6df00b2740 msgr2=0x7f6df00b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:14.080 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.074+0000 7f6df60fa700 1 -- 172.21.15.91:0/2926510876 shutdown_connections 2023-11-24T06:09:14.080 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.074+0000 7f6df60fa700 1 -- 172.21.15.91:0/2926510876 wait complete. 2023-11-24T06:09:14.083 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.075+0000 7f6df60fa700 1 Processor -- start 2023-11-24T06:09:14.083 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.075+0000 7f6df60fa700 1 -- start start 2023-11-24T06:09:14.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6df60fa700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6df006b620 0x7f6df01615d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6df60fa700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6df006ee60 0x7f6df00b0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6df60fa700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6df00b5ab0 0x7f6df00b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6df60fa700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df013f990 con 0x7f6df006ee60 2023-11-24T06:09:14.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6df60fa700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df01f7c10 con 0x7f6df00b5ab0 2023-11-24T06:09:14.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6df60fa700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6df013f060 con 0x7f6df006b620 2023-11-24T06:09:14.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6de6ffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6df006ee60 0x7f6df00b0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:14.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6de6ffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6df006ee60 0x7f6df00b0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59968/0 (socket says 172.21.15.91:59968) 2023-11-24T06:09:14.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6de6ffd700 1 -- 172.21.15.91:0/1790178885 learned_addr learned my addr 172.21.15.91:0/1790178885 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:14.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6def7fe700 1 --2- 172.21.15.91:0/1790178885 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6df006b620 0x7f6df01615d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:14.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6deffff700 1 --2- 172.21.15.91:0/1790178885 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6df00b5ab0 0x7f6df00b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:14.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6de6ffd700 1 -- 172.21.15.91:0/1790178885 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6df006b620 msgr2=0x7f6df01615d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6de6ffd700 1 --2- 172.21.15.91:0/1790178885 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6df006b620 0x7f6df01615d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.086 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6de6ffd700 1 -- 172.21.15.91:0/1790178885 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6df00b5ab0 msgr2=0x7f6df00b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.086 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6de6ffd700 1 --2- 172.21.15.91:0/1790178885 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6df00b5ab0 0x7f6df00b10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.086 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6de6ffd700 1 -- 172.21.15.91:0/1790178885 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de00094a0 con 0x7f6df006ee60 2023-11-24T06:09:14.086 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6deffff700 1 --2- 172.21.15.91:0/1790178885 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6df00b5ab0 0x7f6df00b10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:14.086 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6de6ffd700 1 --2- 172.21.15.91:0/1790178885 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6df006ee60 0x7f6df00b0ba0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6dd80011e0 tx=0x7f6dd800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:14.086 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6decff9700 1 -- 172.21.15.91:0/1790178885 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd8004c80 con 0x7f6df006ee60 2023-11-24T06:09:14.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6decff9700 1 -- 172.21.15.91:0/1790178885 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd8004de0 con 0x7f6df006ee60 2023-11-24T06:09:14.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6df60fa700 1 -- 172.21.15.91:0/1790178885 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df013f340 con 0x7f6df006ee60 2023-11-24T06:09:14.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6decff9700 1 -- 172.21.15.91:0/1790178885 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd8007310 con 0x7f6df006ee60 2023-11-24T06:09:14.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.076+0000 7f6df60fa700 1 -- 172.21.15.91:0/1790178885 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df0161b10 con 0x7f6df006ee60 2023-11-24T06:09:14.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.077+0000 7f6df60fa700 1 -- 172.21.15.91:0/1790178885 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df0075480 con 0x7f6df006ee60 2023-11-24T06:09:14.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.078+0000 7f6decff9700 1 -- 172.21.15.91:0/1790178885 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f6dd8007470 con 0x7f6df006ee60 2023-11-24T06:09:14.087 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.078+0000 7f6decff9700 1 --2- 172.21.15.91:0/1790178885 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6dd006af60 0x7f6dd006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.078+0000 7f6decff9700 1 -- 172.21.15.91:0/1790178885 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6dd80893f0 con 0x7f6df006ee60 2023-11-24T06:09:14.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.078+0000 7f6def7fe700 1 --2- 172.21.15.91:0/1790178885 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6dd006af60 0x7f6dd006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:14.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.079+0000 7f6def7fe700 1 --2- 172.21.15.91:0/1790178885 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6dd006af60 0x7f6dd006d420 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6de0000c00 tx=0x7f6de0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:14.088 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.082+0000 7f6decff9700 1 -- 172.21.15.91:0/1790178885 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6dd80544b0 con 0x7f6df006ee60 2023-11-24T06:09:14.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.272+0000 7f6df60fa700 1 -- 172.21.15.91:0/1790178885 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6df013fad0 con 0x7f6df006ee60 2023-11-24T06:09:14.275 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.272+0000 7f6decff9700 1 -- 172.21.15.91:0/1790178885 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6dd8057ad0 con 0x7f6df006ee60 2023-11-24T06:09:14.275 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 32 2023-11-24T06:09:14.277 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.274+0000 7f6df60fa700 1 -- 172.21.15.91:0/1790178885 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6dd006af60 msgr2=0x7f6dd006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.278 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.274+0000 7f6df60fa700 1 --2- 172.21.15.91:0/1790178885 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6dd006af60 0x7f6dd006d420 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6de0000c00 tx=0x7f6de0008030 comp rx=0 tx=0).stop 2023-11-24T06:09:14.278 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.274+0000 7f6df60fa700 1 -- 172.21.15.91:0/1790178885 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6df006ee60 msgr2=0x7f6df00b0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.278 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.274+0000 7f6df60fa700 1 --2- 172.21.15.91:0/1790178885 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6df006ee60 0x7f6df00b0ba0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6dd80011e0 tx=0x7f6dd800c8b0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.278 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.274+0000 7f6df60fa700 1 -- 172.21.15.91:0/1790178885 shutdown_connections 2023-11-24T06:09:14.278 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.274+0000 7f6df60fa700 1 --2- 172.21.15.91:0/1790178885 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6dd006af60 0x7f6dd006d420 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.278 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.274+0000 7f6df60fa700 1 --2- 172.21.15.91:0/1790178885 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6df00b5ab0 0x7f6df00b10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.274+0000 7f6df60fa700 1 --2- 172.21.15.91:0/1790178885 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6df006b620 0x7f6df01615d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.274+0000 7f6df60fa700 1 --2- 172.21.15.91:0/1790178885 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6df006ee60 0x7f6df00b0ba0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.274+0000 7f6df60fa700 1 -- 172.21.15.91:0/1790178885 >> 172.21.15.91:0/1790178885 conn(0x7f6df00b2740 msgr2=0x7f6df0145fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:14.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.274+0000 7f6df60fa700 1 -- 172.21.15.91:0/1790178885 shutdown_connections 2023-11-24T06:09:14.279 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.274+0000 7f6df60fa700 1 -- 172.21.15.91:0/1790178885 wait complete. 2023-11-24T06:09:14.289 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:09:14.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.410+0000 7f7f3e3c1700 1 Processor -- start 2023-11-24T06:09:14.413 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.410+0000 7f7f3e3c1700 1 -- start start 2023-11-24T06:09:14.414 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.411+0000 7f7f3e3c1700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7f38164f40 0x7f7f38167330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.414 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.411+0000 7f7f3e3c1700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f380b14c0 con 0x7f7f380b4240 2023-11-24T06:09:14.415 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.411+0000 7f7f3e3c1700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f38160a40 con 0x7f7f3815f410 2023-11-24T06:09:14.415 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.411+0000 7f7f3e3c1700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f380b46f0 con 0x7f7f38164f40 2023-11-24T06:09:14.415 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.411+0000 7f7f377fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f7f380b4240 0x7f7f38160330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54910/0 (socket says 172.21.15.91:54910) 2023-11-24T06:09:14.415 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.411+0000 7f7f377fe700 1 -- 172.21.15.91:0/3046609966 learned_addr learned my addr 172.21.15.91:0/3046609966 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:14.415 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.411+0000 7f7f3c95e700 1 --2- 172.21.15.91:0/3046609966 >> v2:172.21.15.94:3300/0 conn(0x7f7f38164f40 0x7f7f38167330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:14.415 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.411+0000 7f7f36ffd700 1 -- 172.21.15.91:0/3046609966 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563803068 0 0) 0x7f7f380b14c0 con 0x7f7f380b4240 2023-11-24T06:09:14.415 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.411+0000 7f7f36ffd700 1 -- 172.21.15.91:0/3046609966 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f200035b0 con 0x7f7f380b4240 2023-11-24T06:09:14.416 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.411+0000 7f7f3c95e700 1 -- 172.21.15.91:0/3046609966 >> v1:172.21.15.94:6789/0 conn(0x7f7f3815f410 legacy=0x7f7f3815fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.416 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.411+0000 7f7f3c95e700 1 -- 172.21.15.91:0/3046609966 >> v1:172.21.15.91:6789/0 conn(0x7f7f380b4240 legacy=0x7f7f38160330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.416 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.411+0000 7f7f3c95e700 1 -- 172.21.15.91:0/3046609966 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f380b3270 con 0x7f7f38164f40 2023-11-24T06:09:14.416 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.411+0000 7f7f3c95e700 1 --2- 172.21.15.91:0/3046609966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f38164f40 0x7f7f38167330 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009d90 tx=0x7f7f3000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=526d8af261c59cbf server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:14.416 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.412+0000 7f7f36ffd700 1 -- 172.21.15.91:0/3046609966 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3000e070 con 0x7f7f38164f40 2023-11-24T06:09:14.416 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.412+0000 7f7f36ffd700 1 -- 172.21.15.91:0/3046609966 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3000bb20 con 0x7f7f38164f40 2023-11-24T06:09:14.416 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.412+0000 7f7f36ffd700 1 -- 172.21.15.91:0/3046609966 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f30012430 con 0x7f7f38164f40 2023-11-24T06:09:14.420 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.412+0000 7f7f3e3c1700 1 -- 172.21.15.91:0/3046609966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f38164f40 msgr2=0x7f7f38167330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.420 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.412+0000 7f7f3e3c1700 1 --2- 172.21.15.91:0/3046609966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f38164f40 0x7f7f38167330 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009d90 tx=0x7f7f3000b330 comp rx=0 tx=0).stop 2023-11-24T06:09:14.420 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.412+0000 7f7f3e3c1700 1 -- 172.21.15.91:0/3046609966 shutdown_connections 2023-11-24T06:09:14.420 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.412+0000 7f7f3e3c1700 1 --2- 172.21.15.91:0/3046609966 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f38164f40 0x7f7f38167330 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.420 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.412+0000 7f7f3e3c1700 1 -- 172.21.15.91:0/3046609966 >> 172.21.15.91:0/3046609966 conn(0x7f7f380b3720 msgr2=0x7f7f380b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:14.421 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.412+0000 7f7f3e3c1700 1 -- 172.21.15.91:0/3046609966 shutdown_connections 2023-11-24T06:09:14.421 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.412+0000 7f7f3e3c1700 1 -- 172.21.15.91:0/3046609966 wait complete. 2023-11-24T06:09:14.421 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.413+0000 7f7f3e3c1700 1 Processor -- start 2023-11-24T06:09:14.421 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.414+0000 7f7f3e3c1700 1 -- start start 2023-11-24T06:09:14.421 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.414+0000 7f7f3e3c1700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7f380b4240 0x7f7f380af410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.421 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.414+0000 7f7f3e3c1700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f3815f410 0x7f7f3815df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.421 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.414+0000 7f7f3e3c1700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f38164f40 0x7f7f3815e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.414+0000 7f7f3e3c1700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f381f3ad0 con 0x7f7f38164f40 2023-11-24T06:09:14.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.414+0000 7f7f3e3c1700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f380af950 con 0x7f7f3815f410 2023-11-24T06:09:14.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.414+0000 7f7f3e3c1700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3806c0e0 con 0x7f7f380b4240 2023-11-24T06:09:14.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.414+0000 7f7f3c95e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f38164f40 0x7f7f3815e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:14.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.414+0000 7f7f3c95e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f38164f40 0x7f7f3815e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59984/0 (socket says 172.21.15.91:59984) 2023-11-24T06:09:14.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.414+0000 7f7f3c95e700 1 -- 172.21.15.91:0/1444809759 learned_addr learned my addr 172.21.15.91:0/1444809759 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:14.422 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.415+0000 7f7f37fff700 1 --2- 172.21.15.91:0/1444809759 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7f380b4240 0x7f7f380af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:14.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.415+0000 7f7f377fe700 1 --2- 172.21.15.91:0/1444809759 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f3815f410 0x7f7f3815df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:14.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.415+0000 7f7f3c95e700 1 -- 172.21.15.91:0/1444809759 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7f380b4240 msgr2=0x7f7f380af410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.415+0000 7f7f3c95e700 1 --2- 172.21.15.91:0/1444809759 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7f380b4240 0x7f7f380af410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.415+0000 7f7f3c95e700 1 -- 172.21.15.91:0/1444809759 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f3815f410 msgr2=0x7f7f3815df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.415+0000 7f7f3c95e700 1 --2- 172.21.15.91:0/1444809759 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f3815f410 0x7f7f3815df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.415+0000 7f7f3c95e700 1 -- 172.21.15.91:0/1444809759 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f30009a70 con 0x7f7f38164f40 2023-11-24T06:09:14.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.415+0000 7f7f37fff700 1 --2- 172.21.15.91:0/1444809759 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7f380b4240 0x7f7f380af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:14.424 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.415+0000 7f7f377fe700 1 --2- 172.21.15.91:0/1444809759 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f3815f410 0x7f7f3815df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:14.424 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.415+0000 7f7f3c95e700 1 --2- 172.21.15.91:0/1444809759 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f38164f40 0x7f7f3815e4b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7f38072590 tx=0x7f7f30004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:14.424 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.415+0000 7f7f34ff9700 1 -- 172.21.15.91:0/1444809759 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f30012430 con 0x7f7f38164f40 2023-11-24T06:09:14.424 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.415+0000 7f7f34ff9700 1 -- 172.21.15.91:0/1444809759 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f30020070 con 0x7f7f38164f40 2023-11-24T06:09:14.424 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.415+0000 7f7f3e3c1700 1 -- 172.21.15.91:0/1444809759 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f380aeee0 con 0x7f7f38164f40 2023-11-24T06:09:14.424 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.415+0000 7f7f34ff9700 1 -- 172.21.15.91:0/1444809759 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f30004280 con 0x7f7f38164f40 2023-11-24T06:09:14.424 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.415+0000 7f7f3e3c1700 1 -- 172.21.15.91:0/1444809759 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3806c330 con 0x7f7f38164f40 2023-11-24T06:09:14.425 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.416+0000 7f7f3e3c1700 1 -- 172.21.15.91:0/1444809759 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f38075480 con 0x7f7f38164f40 2023-11-24T06:09:14.425 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.416+0000 7f7f34ff9700 1 -- 172.21.15.91:0/1444809759 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f7f300043e0 con 0x7f7f38164f40 2023-11-24T06:09:14.425 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.417+0000 7f7f34ff9700 1 --2- 172.21.15.91:0/1444809759 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7f2406af60 0x7f7f2406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.425 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.417+0000 7f7f37fff700 1 --2- 172.21.15.91:0/1444809759 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7f2406af60 0x7f7f2406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:14.425 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.417+0000 7f7f34ff9700 1 -- 172.21.15.91:0/1444809759 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f30022070 con 0x7f7f38164f40 2023-11-24T06:09:14.425 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.417+0000 7f7f37fff700 1 --2- 172.21.15.91:0/1444809759 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7f2406af60 0x7f7f2406d420 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7f28000f60 tx=0x7f7f28009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:14.426 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.421+0000 7f7f34ff9700 1 -- 172.21.15.91:0/1444809759 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f30056830 con 0x7f7f38164f40 2023-11-24T06:09:14.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.615+0000 7f7f3e3c1700 1 -- 172.21.15.91:0/1444809759 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7f380af1f0 con 0x7f7f38164f40 2023-11-24T06:09:14.618 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.616+0000 7f7f34ff9700 1 -- 172.21.15.91:0/1444809759 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7f3001ae60 con 0x7f7f38164f40 2023-11-24T06:09:14.619 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 64 2023-11-24T06:09:14.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.617+0000 7f7f3e3c1700 1 -- 172.21.15.91:0/1444809759 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7f2406af60 msgr2=0x7f7f2406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.617+0000 7f7f3e3c1700 1 --2- 172.21.15.91:0/1444809759 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7f2406af60 0x7f7f2406d420 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7f28000f60 tx=0x7f7f28009040 comp rx=0 tx=0).stop 2023-11-24T06:09:14.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.617+0000 7f7f3e3c1700 1 -- 172.21.15.91:0/1444809759 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f38164f40 msgr2=0x7f7f3815e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.617+0000 7f7f3e3c1700 1 --2- 172.21.15.91:0/1444809759 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f38164f40 0x7f7f3815e4b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7f38072590 tx=0x7f7f30004ad0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.618+0000 7f7f3e3c1700 1 -- 172.21.15.91:0/1444809759 shutdown_connections 2023-11-24T06:09:14.621 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.618+0000 7f7f3e3c1700 1 --2- 172.21.15.91:0/1444809759 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7f2406af60 0x7f7f2406d420 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.618+0000 7f7f3e3c1700 1 --2- 172.21.15.91:0/1444809759 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7f38164f40 0x7f7f3815e4b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.618+0000 7f7f3e3c1700 1 --2- 172.21.15.91:0/1444809759 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7f380b4240 0x7f7f380af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.618+0000 7f7f3e3c1700 1 --2- 172.21.15.91:0/1444809759 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7f3815f410 0x7f7f3815df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.618+0000 7f7f3e3c1700 1 -- 172.21.15.91:0/1444809759 >> 172.21.15.91:0/1444809759 conn(0x7f7f380b3720 msgr2=0x7f7f380b3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:14.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.618+0000 7f7f3e3c1700 1 -- 172.21.15.91:0/1444809759 shutdown_connections 2023-11-24T06:09:14.622 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.618+0000 7f7f3e3c1700 1 -- 172.21.15.91:0/1444809759 wait complete. 2023-11-24T06:09:14.633 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T06:09:14.754 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.751+0000 7f50e2a5f700 1 Processor -- start 2023-11-24T06:09:14.754 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.751+0000 7f50e2a5f700 1 -- start start 2023-11-24T06:09:14.756 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.752+0000 7f50e2a5f700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f50dc15dfc0 0x7f50dc0d4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.756 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.752+0000 7f50e2a5f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f50dc0d1440 0x7f50dc0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.757 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.752+0000 7f50e2a5f700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f50dc15e3a0 0x7f50dc0d0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.757 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.752+0000 7f50e2a5f700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50dc0d2790 con 0x7f50dc15dfc0 2023-11-24T06:09:14.757 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.752+0000 7f50e2a5f700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50dc0d2a70 con 0x7f50dc15e3a0 2023-11-24T06:09:14.757 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.752+0000 7f50e2a5f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc0c7a30 con 0x7f50dc0d1440 2023-11-24T06:09:14.757 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.752+0000 7f50dbfff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f50dc15dfc0 0x7f50dc0d4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:14.757 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.752+0000 7f50dbfff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f50dc15dfc0 0x7f50dc0d4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59988/0 (socket says 172.21.15.91:59988) 2023-11-24T06:09:14.757 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.752+0000 7f50dbfff700 1 -- 172.21.15.91:0/3975918836 learned_addr learned my addr 172.21.15.91:0/3975918836 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:14.757 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.752+0000 7f50dbfff700 1 -- 172.21.15.91:0/3975918836 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f50dc0d1440 msgr2=0x7f50dc0d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:14.757 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.752+0000 7f50dbfff700 1 --2- 172.21.15.91:0/3975918836 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f50dc0d1440 0x7f50dc0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.757 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.752+0000 7f50dbfff700 1 -- 172.21.15.91:0/3975918836 >> v2:172.21.15.94:3300/0 conn(0x7f50dc15e3a0 msgr2=0x7f50dc0d0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:14.757 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.752+0000 7f50dbfff700 1 --2- 172.21.15.91:0/3975918836 >> v2:172.21.15.94:3300/0 conn(0x7f50dc15e3a0 0x7f50dc0d0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.752+0000 7f50dbfff700 1 -- 172.21.15.91:0/3975918836 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50dc0c7d10 con 0x7f50dc15dfc0 2023-11-24T06:09:14.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.752+0000 7f50dbfff700 1 --2- 172.21.15.91:0/3975918836 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f50dc15dfc0 0x7f50dc0d4960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f50dc08c290 tx=0x7f50c4009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f667b5914be7bf72 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:14.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.753+0000 7f50daffd700 1 -- 172.21.15.91:0/3975918836 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c400b870 con 0x7f50dc15dfc0 2023-11-24T06:09:14.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.753+0000 7f50daffd700 1 -- 172.21.15.91:0/3975918836 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c400b9d0 con 0x7f50dc15dfc0 2023-11-24T06:09:14.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.753+0000 7f50daffd700 1 -- 172.21.15.91:0/3975918836 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c400bd60 con 0x7f50dc15dfc0 2023-11-24T06:09:14.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.753+0000 7f50e2a5f700 1 -- 172.21.15.91:0/3975918836 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f50dc15dfc0 msgr2=0x7f50dc0d4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.753+0000 7f50e2a5f700 1 --2- 172.21.15.91:0/3975918836 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f50dc15dfc0 0x7f50dc0d4960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f50dc08c290 tx=0x7f50c4009bd0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.753+0000 7f50e2a5f700 1 -- 172.21.15.91:0/3975918836 shutdown_connections 2023-11-24T06:09:14.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.753+0000 7f50e2a5f700 1 --2- 172.21.15.91:0/3975918836 >> v2:172.21.15.94:3300/0 conn(0x7f50dc15e3a0 0x7f50dc0d0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.753+0000 7f50e2a5f700 1 --2- 172.21.15.91:0/3975918836 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f50dc15dfc0 0x7f50dc0d4960 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.753+0000 7f50e2a5f700 1 --2- 172.21.15.91:0/3975918836 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f50dc0d1440 0x7f50dc0d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.758 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.753+0000 7f50e2a5f700 1 -- 172.21.15.91:0/3975918836 >> 172.21.15.91:0/3975918836 conn(0x7f50dc0d3e40 msgr2=0x7f50dc0d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:14.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.753+0000 7f50e2a5f700 1 -- 172.21.15.91:0/3975918836 shutdown_connections 2023-11-24T06:09:14.759 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.753+0000 7f50e2a5f700 1 -- 172.21.15.91:0/3975918836 wait complete. 2023-11-24T06:09:14.761 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.754+0000 7f50e2a5f700 1 Processor -- start 2023-11-24T06:09:14.761 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.755+0000 7f50e2a5f700 1 -- start start 2023-11-24T06:09:14.761 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.755+0000 7f50e2a5f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f50dc0d1440 0x7f50dc15a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.761 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.755+0000 7f50e2a5f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50dc15dfc0 0x7f50dc0cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.762 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.755+0000 7f50e2a5f700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f50dc15e3a0 0x7f50dc159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.762 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.755+0000 7f50e2a5f700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc0ce160 con 0x7f50dc15e3a0 2023-11-24T06:09:14.762 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.755+0000 7f50e2a5f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc15a990 con 0x7f50dc15dfc0 2023-11-24T06:09:14.762 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.755+0000 7f50e2a5f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc0b82f0 con 0x7f50dc0d1440 2023-11-24T06:09:14.762 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.755+0000 7f50e0ffc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f50dc15e3a0 0x7f50dc159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:14.762 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.755+0000 7f50e0ffc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f50dc15e3a0 0x7f50dc159960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:59998/0 (socket says 172.21.15.91:59998) 2023-11-24T06:09:14.762 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.755+0000 7f50e0ffc700 1 -- 172.21.15.91:0/2307815184 learned_addr learned my addr 172.21.15.91:0/2307815184 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:14.762 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.756+0000 7f50dbfff700 1 --2- 172.21.15.91:0/2307815184 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f50dc0d1440 0x7f50dc15a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:14.762 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.756+0000 7f50db7fe700 1 --2- 172.21.15.91:0/2307815184 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50dc15dfc0 0x7f50dc0cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:14.762 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.756+0000 7f50e0ffc700 1 -- 172.21.15.91:0/2307815184 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f50dc0d1440 msgr2=0x7f50dc15a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.762 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.756+0000 7f50e0ffc700 1 --2- 172.21.15.91:0/2307815184 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f50dc0d1440 0x7f50dc15a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.762 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.756+0000 7f50e0ffc700 1 -- 172.21.15.91:0/2307815184 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50dc15dfc0 msgr2=0x7f50dc0cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.763 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.756+0000 7f50e0ffc700 1 --2- 172.21.15.91:0/2307815184 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50dc15dfc0 0x7f50dc0cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.763 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.756+0000 7f50e0ffc700 1 -- 172.21.15.91:0/2307815184 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c4009550 con 0x7f50dc15e3a0 2023-11-24T06:09:14.763 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.756+0000 7f50e0ffc700 1 --2- 172.21.15.91:0/2307815184 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f50dc15e3a0 0x7f50dc159960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f50d400b9e0 tx=0x7f50d400bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:14.763 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.756+0000 7f50d8ff9700 1 -- 172.21.15.91:0/2307815184 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d400d8e0 con 0x7f50dc15e3a0 2023-11-24T06:09:14.763 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.756+0000 7f50d8ff9700 1 -- 172.21.15.91:0/2307815184 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d400da60 con 0x7f50dc15e3a0 2023-11-24T06:09:14.763 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.756+0000 7f50d8ff9700 1 -- 172.21.15.91:0/2307815184 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d400de10 con 0x7f50dc15e3a0 2023-11-24T06:09:14.763 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.756+0000 7f50e2a5f700 1 -- 172.21.15.91:0/2307815184 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50dc0b85d0 con 0x7f50dc15e3a0 2023-11-24T06:09:14.763 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.756+0000 7f50e2a5f700 1 -- 172.21.15.91:0/2307815184 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50dc0b8b90 con 0x7f50dc15e3a0 2023-11-24T06:09:14.763 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.757+0000 7f50e2a5f700 1 -- 172.21.15.91:0/2307815184 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50dc068670 con 0x7f50dc15e3a0 2023-11-24T06:09:14.763 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.757+0000 7f50d8ff9700 1 -- 172.21.15.91:0/2307815184 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f50d4015420 con 0x7f50dc15e3a0 2023-11-24T06:09:14.763 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.758+0000 7f50d8ff9700 1 --2- 172.21.15.91:0/2307815184 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f50bc06af60 0x7f50bc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:14.763 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.758+0000 7f50d8ff9700 1 -- 172.21.15.91:0/2307815184 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50d4011070 con 0x7f50dc15e3a0 2023-11-24T06:09:14.763 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.758+0000 7f50dbfff700 1 --2- 172.21.15.91:0/2307815184 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f50bc06af60 0x7f50bc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:14.764 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.758+0000 7f50dbfff700 1 --2- 172.21.15.91:0/2307815184 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f50bc06af60 0x7f50bc06d420 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f50c4000c00 tx=0x7f50c4008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:14.765 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.762+0000 7f50d8ff9700 1 -- 172.21.15.91:0/2307815184 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50d404ee90 con 0x7f50dc15e3a0 2023-11-24T06:09:14.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.968+0000 7f50e2a5f700 1 -- 172.21.15.91:0/2307815184 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f50dc1615c0 con 0x7f50dc15e3a0 2023-11-24T06:09:14.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.969+0000 7f50d8ff9700 1 -- 172.21.15.91:0/2307815184 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7f50d4054520 con 0x7f50dc15e3a0 2023-11-24T06:09:14.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.971+0000 7f50e2a5f700 1 -- 172.21.15.91:0/2307815184 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f50bc06af60 msgr2=0x7f50bc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.971+0000 7f50e2a5f700 1 --2- 172.21.15.91:0/2307815184 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f50bc06af60 0x7f50bc06d420 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f50c4000c00 tx=0x7f50c4008030 comp rx=0 tx=0).stop 2023-11-24T06:09:14.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.971+0000 7f50e2a5f700 1 -- 172.21.15.91:0/2307815184 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f50dc15e3a0 msgr2=0x7f50dc159960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:14.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.971+0000 7f50e2a5f700 1 --2- 172.21.15.91:0/2307815184 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f50dc15e3a0 0x7f50dc159960 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f50d400b9e0 tx=0x7f50d400bcf0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.971+0000 7f50e2a5f700 1 -- 172.21.15.91:0/2307815184 shutdown_connections 2023-11-24T06:09:14.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.971+0000 7f50e2a5f700 1 --2- 172.21.15.91:0/2307815184 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f50bc06af60 0x7f50bc06d420 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.971+0000 7f50e2a5f700 1 --2- 172.21.15.91:0/2307815184 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f50dc15e3a0 0x7f50dc159960 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.974 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.971+0000 7f50e2a5f700 1 --2- 172.21.15.91:0/2307815184 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f50dc0d1440 0x7f50dc15a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.975 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.971+0000 7f50e2a5f700 1 --2- 172.21.15.91:0/2307815184 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f50dc15dfc0 0x7f50dc0cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:14.975 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.971+0000 7f50e2a5f700 1 -- 172.21.15.91:0/2307815184 >> 172.21.15.91:0/2307815184 conn(0x7f50dc0d3e40 msgr2=0x7f50dc06b960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:14.975 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.971+0000 7f50e2a5f700 1 -- 172.21.15.91:0/2307815184 shutdown_connections 2023-11-24T06:09:14.975 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:14.971+0000 7f50e2a5f700 1 -- 172.21.15.91:0/2307815184 wait complete. 2023-11-24T06:09:14.975 INFO:teuthology.orchestra.run.smithi091.stderr:listed 0 entries 2023-11-24T06:09:14.985 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-11-24T06:09:14.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:14.986 DEBUG:teuthology.orchestra.run.smithi091:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-11-24T06:09:15.026 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi091.front.sepia.ceph.com 2023-11-24T06:09:15.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:15.027 DEBUG:teuthology.orchestra.run.smithi091:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-11-24T06:09:15.083 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:09:15.083 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:09:15.104 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:09:15.104 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:09:15.163 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:09:15.284 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.281+0000 7f531f798700 1 Processor -- start 2023-11-24T06:09:15.284 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.281+0000 7f531f798700 1 -- start start 2023-11-24T06:09:15.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.282+0000 7f531f798700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f53180cf2b0 0x7f531815c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.282+0000 7f531f798700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53180ce5c0 0x7f53180cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.282+0000 7f531f798700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53180d0a90 con 0x7f531815bde0 2023-11-24T06:09:15.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.282+0000 7f531f798700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53180cefb0 con 0x7f53180cf2b0 2023-11-24T06:09:15.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.282+0000 7f531f798700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531806ee10 con 0x7f53180ce5c0 2023-11-24T06:09:15.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.282+0000 7f531cd33700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f53180cf2b0 0x7f531815c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.282+0000 7f531cd33700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f53180cf2b0 0x7f531815c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60004/0 (socket says 172.21.15.91:60004) 2023-11-24T06:09:15.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.282+0000 7f531cd33700 1 -- 172.21.15.91:0/3268542669 learned_addr learned my addr 172.21.15.91:0/3268542669 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:15.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.282+0000 7f531dd35700 1 --2- 172.21.15.91:0/3268542669 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53180ce5c0 0x7f53180cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.287 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.282+0000 7f531cd33700 1 -- 172.21.15.91:0/3268542669 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53180ce5c0 msgr2=0x7f53180cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:15.287 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.282+0000 7f531cd33700 1 --2- 172.21.15.91:0/3268542669 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53180ce5c0 0x7f53180cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.287 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.282+0000 7f531cd33700 1 -- 172.21.15.91:0/3268542669 >> v1:172.21.15.94:6789/0 conn(0x7f531815bde0 legacy=0x7f53180d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:15.287 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.282+0000 7f531cd33700 1 -- 172.21.15.91:0/3268542669 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531806f0f0 con 0x7f53180cf2b0 2023-11-24T06:09:15.287 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.282+0000 7f531cd33700 1 --2- 172.21.15.91:0/3268542669 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53180cf2b0 0x7f531815c1c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f53140098f0 tx=0x7f5314009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c4d219e469c77c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:15.287 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.283+0000 7f530ffff700 1 -- 172.21.15.91:0/3268542669 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5314004230 con 0x7f53180cf2b0 2023-11-24T06:09:15.287 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.283+0000 7f530ffff700 1 -- 172.21.15.91:0/3268542669 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5314004390 con 0x7f53180cf2b0 2023-11-24T06:09:15.287 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.283+0000 7f530ffff700 1 -- 172.21.15.91:0/3268542669 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53140036a0 con 0x7f53180cf2b0 2023-11-24T06:09:15.287 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.283+0000 7f531f798700 1 -- 172.21.15.91:0/3268542669 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53180cf2b0 msgr2=0x7f531815c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:15.287 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.283+0000 7f531f798700 1 --2- 172.21.15.91:0/3268542669 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53180cf2b0 0x7f531815c1c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f53140098f0 tx=0x7f5314009c00 comp rx=0 tx=0).stop 2023-11-24T06:09:15.287 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.283+0000 7f531f798700 1 -- 172.21.15.91:0/3268542669 shutdown_connections 2023-11-24T06:09:15.287 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.283+0000 7f531f798700 1 --2- 172.21.15.91:0/3268542669 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53180ce5c0 0x7f53180cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.287 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.283+0000 7f531f798700 1 --2- 172.21.15.91:0/3268542669 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53180cf2b0 0x7f531815c1c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.288 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.283+0000 7f531f798700 1 -- 172.21.15.91:0/3268542669 >> 172.21.15.91:0/3268542669 conn(0x7f53180b3720 msgr2=0x7f53180b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:15.288 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.283+0000 7f531f798700 1 -- 172.21.15.91:0/3268542669 shutdown_connections 2023-11-24T06:09:15.288 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.283+0000 7f531f798700 1 -- 172.21.15.91:0/3268542669 wait complete. 2023-11-24T06:09:15.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.284+0000 7f531f798700 1 Processor -- start 2023-11-24T06:09:15.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.284+0000 7f531f798700 1 -- start start 2023-11-24T06:09:15.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531f798700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53180ce5c0 0x7f53180b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531f798700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53180cf2b0 0x7f53180b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531f798700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f531815bde0 0x7f53180b5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531f798700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53181539c0 con 0x7f53180cf2b0 2023-11-24T06:09:15.291 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531f798700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53180c6660 con 0x7f53180ce5c0 2023-11-24T06:09:15.291 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531f798700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53180b4b40 con 0x7f531815bde0 2023-11-24T06:09:15.291 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531cd33700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53180cf2b0 0x7f53180b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.291 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531cd33700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53180cf2b0 0x7f53180b2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60010/0 (socket says 172.21.15.91:60010) 2023-11-24T06:09:15.291 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531cd33700 1 -- 172.21.15.91:0/3582404129 learned_addr learned my addr 172.21.15.91:0/3582404129 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:15.291 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531dd35700 1 --2- 172.21.15.91:0/3582404129 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f531815bde0 0x7f53180b5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.291 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531d534700 1 --2- 172.21.15.91:0/3582404129 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53180ce5c0 0x7f53180b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.291 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531cd33700 1 -- 172.21.15.91:0/3582404129 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f531815bde0 msgr2=0x7f53180b5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:15.291 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531cd33700 1 --2- 172.21.15.91:0/3582404129 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f531815bde0 0x7f53180b5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.291 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531cd33700 1 -- 172.21.15.91:0/3582404129 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53180ce5c0 msgr2=0x7f53180b29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:15.291 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531cd33700 1 --2- 172.21.15.91:0/3582404129 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53180ce5c0 0x7f53180b29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.291 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531cd33700 1 -- 172.21.15.91:0/3582404129 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5314009550 con 0x7f53180cf2b0 2023-11-24T06:09:15.291 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.285+0000 7f531dd35700 1 --2- 172.21.15.91:0/3582404129 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f531815bde0 0x7f53180b5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:15.292 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.286+0000 7f531cd33700 1 --2- 172.21.15.91:0/3582404129 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53180cf2b0 0x7f53180b2f30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5314004000 tx=0x7f5314004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:15.292 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.286+0000 7f530dffb700 1 -- 172.21.15.91:0/3582404129 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5314003c80 con 0x7f53180cf2b0 2023-11-24T06:09:15.292 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.286+0000 7f530dffb700 1 -- 172.21.15.91:0/3582404129 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f531401f070 con 0x7f53180cf2b0 2023-11-24T06:09:15.292 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.286+0000 7f531f798700 1 -- 172.21.15.91:0/3582404129 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53180c68b0 con 0x7f53180cf2b0 2023-11-24T06:09:15.292 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.286+0000 7f530dffb700 1 -- 172.21.15.91:0/3582404129 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f531400b660 con 0x7f53180cf2b0 2023-11-24T06:09:15.292 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.286+0000 7f531f798700 1 -- 172.21.15.91:0/3582404129 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53180c6e70 con 0x7f53180cf2b0 2023-11-24T06:09:15.292 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.287+0000 7f531f798700 1 -- 172.21.15.91:0/3582404129 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5318068670 con 0x7f53180cf2b0 2023-11-24T06:09:15.292 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.287+0000 7f530dffb700 1 -- 172.21.15.91:0/3582404129 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f5314021030 con 0x7f53180cf2b0 2023-11-24T06:09:15.292 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.288+0000 7f530dffb700 1 --2- 172.21.15.91:0/3582404129 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f52fc06af60 0x7f52fc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.293 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.288+0000 7f530dffb700 1 -- 172.21.15.91:0/3582404129 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f531408ac20 con 0x7f53180cf2b0 2023-11-24T06:09:15.293 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.288+0000 7f531d534700 1 --2- 172.21.15.91:0/3582404129 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f52fc06af60 0x7f52fc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.293 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.288+0000 7f531d534700 1 --2- 172.21.15.91:0/3582404129 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f52fc06af60 0x7f52fc06d420 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5308009960 tx=0x7f53080090e0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:15.295 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.292+0000 7f530dffb700 1 -- 172.21.15.91:0/3582404129 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5314055d70 con 0x7f53180cf2b0 2023-11-24T06:09:15.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.486+0000 7f531f798700 1 -- 172.21.15.91:0/3582404129 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5318068670 con 0x7f53180cf2b0 2023-11-24T06:09:15.490 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.487+0000 7f530dffb700 1 -- 172.21.15.91:0/3582404129 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16485 (secure 0 0 0) 0x7f5314059390 con 0x7f53180cf2b0 2023-11-24T06:09:15.491 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:15.491 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":34,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:09:00.042725+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:08:56.059841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:08:57.408988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:09:15.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.490+0000 7f531f798700 1 -- 172.21.15.91:0/3582404129 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f52fc06af60 msgr2=0x7f52fc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:15.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.490+0000 7f531f798700 1 --2- 172.21.15.91:0/3582404129 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f52fc06af60 0x7f52fc06d420 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5308009960 tx=0x7f53080090e0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.490+0000 7f531f798700 1 -- 172.21.15.91:0/3582404129 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53180cf2b0 msgr2=0x7f53180b2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:15.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.490+0000 7f531f798700 1 --2- 172.21.15.91:0/3582404129 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53180cf2b0 0x7f53180b2f30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5314004000 tx=0x7f5314004750 comp rx=0 tx=0).stop 2023-11-24T06:09:15.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.490+0000 7f531f798700 1 -- 172.21.15.91:0/3582404129 shutdown_connections 2023-11-24T06:09:15.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.490+0000 7f531f798700 1 --2- 172.21.15.91:0/3582404129 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f52fc06af60 0x7f52fc06d420 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.490+0000 7f531f798700 1 --2- 172.21.15.91:0/3582404129 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53180ce5c0 0x7f53180b29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.490+0000 7f531f798700 1 --2- 172.21.15.91:0/3582404129 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f531815bde0 0x7f53180b5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.490+0000 7f531f798700 1 --2- 172.21.15.91:0/3582404129 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53180cf2b0 0x7f53180b2f30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.490+0000 7f531f798700 1 -- 172.21.15.91:0/3582404129 >> 172.21.15.91:0/3582404129 conn(0x7f53180b3720 msgr2=0x7f53180af280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:15.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.490+0000 7f531f798700 1 -- 172.21.15.91:0/3582404129 shutdown_connections 2023-11-24T06:09:15.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.490+0000 7f531f798700 1 -- 172.21.15.91:0/3582404129 wait complete. 2023-11-24T06:09:15.505 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:08:56.059841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:08:57.408988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:09:15.505 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:09:15.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.625+0000 7f8c3a0e4700 1 Processor -- start 2023-11-24T06:09:15.628 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.625+0000 7f8c3a0e4700 1 -- start start 2023-11-24T06:09:15.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.626+0000 7f8c3a0e4700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f8c340cf2b0 0x7f8c3415c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.626+0000 7f8c3a0e4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c340ce5c0 0x7f8c340cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.630 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.626+0000 7f8c3a0e4700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c340d0a90 con 0x7f8c3415bde0 2023-11-24T06:09:15.631 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.626+0000 7f8c3a0e4700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c340cefb0 con 0x7f8c340cf2b0 2023-11-24T06:09:15.631 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.626+0000 7f8c3a0e4700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3406ee10 con 0x7f8c340ce5c0 2023-11-24T06:09:15.631 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.626+0000 7f8c32ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f8c340cf2b0 0x7f8c3415c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.631 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.626+0000 7f8c337fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f8c3415bde0 0x7f8c340d01b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54950/0 (socket says 172.21.15.91:54950) 2023-11-24T06:09:15.631 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.626+0000 7f8c337fe700 1 -- 172.21.15.91:0/2918041530 learned_addr learned my addr 172.21.15.91:0/2918041530 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:15.631 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.626+0000 7f8c33fff700 1 --2- 172.21.15.91:0/2918041530 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c340ce5c0 0x7f8c340cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.631 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.626+0000 7f8c32ffd700 1 -- 172.21.15.91:0/2918041530 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c340ce5c0 msgr2=0x7f8c340cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:15.631 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.626+0000 7f8c32ffd700 1 --2- 172.21.15.91:0/2918041530 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c340ce5c0 0x7f8c340cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.631 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.626+0000 7f8c32ffd700 1 -- 172.21.15.91:0/2918041530 >> v1:172.21.15.91:6789/0 conn(0x7f8c3415bde0 legacy=0x7f8c340d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:15.631 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.626+0000 7f8c327fc700 1 -- 172.21.15.91:0/2918041530 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684061027 0 0) 0x7f8c340d0a90 con 0x7f8c3415bde0 2023-11-24T06:09:15.631 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.626+0000 7f8c32ffd700 1 -- 172.21.15.91:0/2918041530 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3406f0f0 con 0x7f8c340cf2b0 2023-11-24T06:09:15.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.626+0000 7f8c32ffd700 1 --2- 172.21.15.91:0/2918041530 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8c340cf2b0 0x7f8c3415c1c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8c24009840 tx=0x7f8c24009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c843c1588f90fad server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:15.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.626+0000 7f8c327fc700 1 -- 172.21.15.91:0/2918041530 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c24011830 con 0x7f8c340cf2b0 2023-11-24T06:09:15.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.627+0000 7f8c327fc700 1 -- 172.21.15.91:0/2918041530 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c24011990 con 0x7f8c340cf2b0 2023-11-24T06:09:15.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.627+0000 7f8c327fc700 1 -- 172.21.15.91:0/2918041530 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c24011d20 con 0x7f8c340cf2b0 2023-11-24T06:09:15.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.627+0000 7f8c3a0e4700 1 -- 172.21.15.91:0/2918041530 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8c340cf2b0 msgr2=0x7f8c3415c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:15.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.627+0000 7f8c3a0e4700 1 --2- 172.21.15.91:0/2918041530 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8c340cf2b0 0x7f8c3415c1c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8c24009840 tx=0x7f8c24009b50 comp rx=0 tx=0).stop 2023-11-24T06:09:15.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.627+0000 7f8c3a0e4700 1 -- 172.21.15.91:0/2918041530 shutdown_connections 2023-11-24T06:09:15.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.627+0000 7f8c3a0e4700 1 --2- 172.21.15.91:0/2918041530 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c340ce5c0 0x7f8c340cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.627+0000 7f8c3a0e4700 1 --2- 172.21.15.91:0/2918041530 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8c340cf2b0 0x7f8c3415c1c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.627+0000 7f8c3a0e4700 1 -- 172.21.15.91:0/2918041530 >> 172.21.15.91:0/2918041530 conn(0x7f8c340b3720 msgr2=0x7f8c340b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:15.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.627+0000 7f8c3a0e4700 1 -- 172.21.15.91:0/2918041530 shutdown_connections 2023-11-24T06:09:15.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.627+0000 7f8c3a0e4700 1 -- 172.21.15.91:0/2918041530 wait complete. 2023-11-24T06:09:15.635 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.628+0000 7f8c3a0e4700 1 Processor -- start 2023-11-24T06:09:15.635 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.628+0000 7f8c3a0e4700 1 -- start start 2023-11-24T06:09:15.635 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.629+0000 7f8c3a0e4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c340ce5c0 0x7f8c340c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.635 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.629+0000 7f8c3a0e4700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c340cf2b0 0x7f8c341f4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.635 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.629+0000 7f8c3a0e4700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8c3415bde0 0x7f8c341590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.635 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.629+0000 7f8c3a0e4700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c341f5390 con 0x7f8c3415bde0 2023-11-24T06:09:15.635 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.629+0000 7f8c3a0e4700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c341f55f0 con 0x7f8c340cf2b0 2023-11-24T06:09:15.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.629+0000 7f8c3a0e4700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c341f58d0 con 0x7f8c340ce5c0 2023-11-24T06:09:15.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.629+0000 7f8c33fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8c3415bde0 0x7f8c341590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.629+0000 7f8c33fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8c3415bde0 0x7f8c341590e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60026/0 (socket says 172.21.15.91:60026) 2023-11-24T06:09:15.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.629+0000 7f8c33fff700 1 -- 172.21.15.91:0/1074654454 learned_addr learned my addr 172.21.15.91:0/1074654454 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:15.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.629+0000 7f8c32ffd700 1 --2- 172.21.15.91:0/1074654454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c340cf2b0 0x7f8c341f4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.629+0000 7f8c337fe700 1 --2- 172.21.15.91:0/1074654454 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c340ce5c0 0x7f8c340c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.630+0000 7f8c33fff700 1 -- 172.21.15.91:0/1074654454 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c340ce5c0 msgr2=0x7f8c340c83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:15.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.630+0000 7f8c33fff700 1 --2- 172.21.15.91:0/1074654454 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c340ce5c0 0x7f8c340c83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.630+0000 7f8c33fff700 1 -- 172.21.15.91:0/1074654454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c340cf2b0 msgr2=0x7f8c341f4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:15.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.630+0000 7f8c33fff700 1 --2- 172.21.15.91:0/1074654454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c340cf2b0 0x7f8c341f4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.630+0000 7f8c33fff700 1 -- 172.21.15.91:0/1074654454 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c240094a0 con 0x7f8c3415bde0 2023-11-24T06:09:15.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.630+0000 7f8c33fff700 1 --2- 172.21.15.91:0/1074654454 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8c3415bde0 0x7f8c341590e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c007e40 tx=0x7f8c2c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:15.636 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.630+0000 7f8c1bfff700 1 -- 172.21.15.91:0/1074654454 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c00ed30 con 0x7f8c3415bde0 2023-11-24T06:09:15.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.630+0000 7f8c1bfff700 1 -- 172.21.15.91:0/1074654454 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2c014420 con 0x7f8c3415bde0 2023-11-24T06:09:15.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.630+0000 7f8c1bfff700 1 -- 172.21.15.91:0/1074654454 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c0147d0 con 0x7f8c3415bde0 2023-11-24T06:09:15.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.630+0000 7f8c3a0e4700 1 -- 172.21.15.91:0/1074654454 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c341f5bb0 con 0x7f8c3415bde0 2023-11-24T06:09:15.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.630+0000 7f8c3a0e4700 1 -- 172.21.15.91:0/1074654454 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c341f1820 con 0x7f8c3415bde0 2023-11-24T06:09:15.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.631+0000 7f8c3a0e4700 1 -- 172.21.15.91:0/1074654454 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c34075480 con 0x7f8c3415bde0 2023-11-24T06:09:15.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.631+0000 7f8c1bfff700 1 -- 172.21.15.91:0/1074654454 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f8c2c014930 con 0x7f8c3415bde0 2023-11-24T06:09:15.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.632+0000 7f8c1bfff700 1 --2- 172.21.15.91:0/1074654454 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8c2006af60 0x7f8c2006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.632+0000 7f8c1bfff700 1 -- 172.21.15.91:0/1074654454 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c2c089c20 con 0x7f8c3415bde0 2023-11-24T06:09:15.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.632+0000 7f8c337fe700 1 --2- 172.21.15.91:0/1074654454 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8c2006af60 0x7f8c2006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.632+0000 7f8c337fe700 1 --2- 172.21.15.91:0/1074654454 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8c2006af60 0x7f8c2006d420 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8c28000f60 tx=0x7f8c28007e70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:15.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.636+0000 7f8c1bfff700 1 -- 172.21.15.91:0/1074654454 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8c2c054d70 con 0x7f8c3415bde0 2023-11-24T06:09:15.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.828+0000 7f8c3a0e4700 1 -- 172.21.15.91:0/1074654454 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8c34070410 con 0x7f8c3415bde0 2023-11-24T06:09:15.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.828+0000 7f8c1bfff700 1 -- 172.21.15.91:0/1074654454 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8c2c058390 con 0x7f8c3415bde0 2023-11-24T06:09:15.831 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:09:15.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.830+0000 7f8c3a0e4700 1 -- 172.21.15.91:0/1074654454 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8c2006af60 msgr2=0x7f8c2006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:15.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.830+0000 7f8c3a0e4700 1 --2- 172.21.15.91:0/1074654454 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8c2006af60 0x7f8c2006d420 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8c28000f60 tx=0x7f8c28007e70 comp rx=0 tx=0).stop 2023-11-24T06:09:15.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.830+0000 7f8c3a0e4700 1 -- 172.21.15.91:0/1074654454 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8c3415bde0 msgr2=0x7f8c341590e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:15.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.830+0000 7f8c3a0e4700 1 --2- 172.21.15.91:0/1074654454 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8c3415bde0 0x7f8c341590e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c007e40 tx=0x7f8c2c00e3c0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.830+0000 7f8c3a0e4700 1 -- 172.21.15.91:0/1074654454 shutdown_connections 2023-11-24T06:09:15.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.830+0000 7f8c3a0e4700 1 --2- 172.21.15.91:0/1074654454 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8c2006af60 0x7f8c2006d420 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.830+0000 7f8c3a0e4700 1 --2- 172.21.15.91:0/1074654454 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8c3415bde0 0x7f8c341590e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.830+0000 7f8c3a0e4700 1 --2- 172.21.15.91:0/1074654454 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8c340ce5c0 0x7f8c340c83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.830+0000 7f8c3a0e4700 1 --2- 172.21.15.91:0/1074654454 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8c340cf2b0 0x7f8c341f4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.830+0000 7f8c3a0e4700 1 -- 172.21.15.91:0/1074654454 >> 172.21.15.91:0/1074654454 conn(0x7f8c340b3720 msgr2=0x7f8c3415a000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:15.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.830+0000 7f8c3a0e4700 1 -- 172.21.15.91:0/1074654454 shutdown_connections 2023-11-24T06:09:15.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.830+0000 7f8c3a0e4700 1 -- 172.21.15.91:0/1074654454 wait complete. 2023-11-24T06:09:15.845 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:09:15.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.966+0000 7fe9084d8700 1 Processor -- start 2023-11-24T06:09:15.969 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.966+0000 7fe9084d8700 1 -- start start 2023-11-24T06:09:15.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.967+0000 7fe9084d8700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fe90015a440 0x7fe9000cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.967+0000 7fe9084d8700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fe90015eb80 0x7fe9000d04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.967+0000 7fe9084d8700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9000b05f0 con 0x7fe90015a820 2023-11-24T06:09:15.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.967+0000 7fe9084d8700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe90015f080 con 0x7fe90015eb80 2023-11-24T06:09:15.971 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.967+0000 7fe9084d8700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9000afd50 con 0x7fe90015a440 2023-11-24T06:09:15.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.967+0000 7fe906a75700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fe90015eb80 0x7fe9000d04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.967+0000 7fe905a73700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fe90015a820 0x7fe9000726a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54966/0 (socket says 172.21.15.91:54966) 2023-11-24T06:09:15.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.967+0000 7fe905a73700 1 -- 172.21.15.91:0/2465654796 learned_addr learned my addr 172.21.15.91:0/2465654796 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:15.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.967+0000 7fe906274700 1 --2- 172.21.15.91:0/2465654796 >> v2:172.21.15.94:3300/0 conn(0x7fe90015a440 0x7fe9000cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.967+0000 7fe906a75700 1 -- 172.21.15.91:0/2465654796 >> v2:172.21.15.94:3300/0 conn(0x7fe90015a440 msgr2=0x7fe9000cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:15.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.967+0000 7fe906a75700 1 --2- 172.21.15.91:0/2465654796 >> v2:172.21.15.94:3300/0 conn(0x7fe90015a440 0x7fe9000cffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.967+0000 7fe906a75700 1 -- 172.21.15.91:0/2465654796 >> v1:172.21.15.91:6789/0 conn(0x7fe90015a820 legacy=0x7fe9000726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:15.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.967+0000 7fe905272700 1 -- 172.21.15.91:0/2465654796 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759208200 0 0) 0x7fe9000b05f0 con 0x7fe90015a820 2023-11-24T06:09:15.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.967+0000 7fe906a75700 1 -- 172.21.15.91:0/2465654796 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9000b0030 con 0x7fe90015eb80 2023-11-24T06:09:15.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.967+0000 7fe906a75700 1 --2- 172.21.15.91:0/2465654796 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe90015eb80 0x7fe9000d04e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe900069520 tx=0x7fe8f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b853908a0b37c14 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:15.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.967+0000 7fe905272700 1 -- 172.21.15.91:0/2465654796 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f800e070 con 0x7fe90015eb80 2023-11-24T06:09:15.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.968+0000 7fe905272700 1 -- 172.21.15.91:0/2465654796 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f800bbd0 con 0x7fe90015eb80 2023-11-24T06:09:15.972 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.968+0000 7fe905272700 1 -- 172.21.15.91:0/2465654796 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8012430 con 0x7fe90015eb80 2023-11-24T06:09:15.973 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.968+0000 7fe9084d8700 1 -- 172.21.15.91:0/2465654796 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe90015eb80 msgr2=0x7fe9000d04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:15.973 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.968+0000 7fe9084d8700 1 --2- 172.21.15.91:0/2465654796 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe90015eb80 0x7fe9000d04e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fe900069520 tx=0x7fe8f800b330 comp rx=0 tx=0).stop 2023-11-24T06:09:15.973 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.968+0000 7fe9084d8700 1 -- 172.21.15.91:0/2465654796 shutdown_connections 2023-11-24T06:09:15.973 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.968+0000 7fe9084d8700 1 --2- 172.21.15.91:0/2465654796 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe90015eb80 0x7fe9000d04e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.973 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.968+0000 7fe9084d8700 1 --2- 172.21.15.91:0/2465654796 >> v2:172.21.15.94:3300/0 conn(0x7fe90015a440 0x7fe9000cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.973 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.968+0000 7fe9084d8700 1 -- 172.21.15.91:0/2465654796 >> 172.21.15.91:0/2465654796 conn(0x7fe9000b5070 msgr2=0x7fe9000b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:15.973 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.968+0000 7fe9084d8700 1 -- 172.21.15.91:0/2465654796 shutdown_connections 2023-11-24T06:09:15.973 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.968+0000 7fe9084d8700 1 -- 172.21.15.91:0/2465654796 wait complete. 2023-11-24T06:09:15.976 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.969+0000 7fe9084d8700 1 Processor -- start 2023-11-24T06:09:15.976 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.969+0000 7fe9084d8700 1 -- start start 2023-11-24T06:09:15.976 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.970+0000 7fe9084d8700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe90015a440 0x7fe9000cf000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.976 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.970+0000 7fe9084d8700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe90015eb80 0x7fe9001452c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.976 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.970+0000 7fe9084d8700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe90015ac10 0x7fe900145800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.976 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.970+0000 7fe9084d8700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9001519e0 con 0x7fe90015ac10 2023-11-24T06:09:15.976 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.970+0000 7fe9084d8700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9001537a0 con 0x7fe90015a440 2023-11-24T06:09:15.976 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.970+0000 7fe9084d8700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe900153a80 con 0x7fe90015eb80 2023-11-24T06:09:15.976 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.970+0000 7fe906a75700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe90015ac10 0x7fe900145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.976 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.970+0000 7fe906a75700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe90015ac10 0x7fe900145800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60040/0 (socket says 172.21.15.91:60040) 2023-11-24T06:09:15.977 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.970+0000 7fe906a75700 1 -- 172.21.15.91:0/4240924616 learned_addr learned my addr 172.21.15.91:0/4240924616 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:15.977 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.970+0000 7fe906274700 1 --2- 172.21.15.91:0/4240924616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe90015a440 0x7fe9000cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.977 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.970+0000 7fe905a73700 1 --2- 172.21.15.91:0/4240924616 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe90015eb80 0x7fe9001452c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.977 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.970+0000 7fe906a75700 1 -- 172.21.15.91:0/4240924616 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe90015eb80 msgr2=0x7fe9001452c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:15.977 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.970+0000 7fe906a75700 1 --2- 172.21.15.91:0/4240924616 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe90015eb80 0x7fe9001452c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.977 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.971+0000 7fe906a75700 1 -- 172.21.15.91:0/4240924616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe90015a440 msgr2=0x7fe9000cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:15.977 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.971+0000 7fe906a75700 1 --2- 172.21.15.91:0/4240924616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe90015a440 0x7fe9000cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:15.977 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.971+0000 7fe906a75700 1 -- 172.21.15.91:0/4240924616 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f8009a70 con 0x7fe90015ac10 2023-11-24T06:09:15.977 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.971+0000 7fe906274700 1 --2- 172.21.15.91:0/4240924616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe90015a440 0x7fe9000cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:15.977 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.971+0000 7fe906a75700 1 --2- 172.21.15.91:0/4240924616 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe90015ac10 0x7fe900145800 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8013000 tx=0x7fe8f8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:15.977 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.971+0000 7fe8f6ffd700 1 -- 172.21.15.91:0/4240924616 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8012430 con 0x7fe90015ac10 2023-11-24T06:09:15.977 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.971+0000 7fe8f6ffd700 1 -- 172.21.15.91:0/4240924616 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f8020070 con 0x7fe90015ac10 2023-11-24T06:09:15.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.971+0000 7fe9084d8700 1 -- 172.21.15.91:0/4240924616 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90015b110 con 0x7fe90015ac10 2023-11-24T06:09:15.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.971+0000 7fe8f6ffd700 1 -- 172.21.15.91:0/4240924616 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8004280 con 0x7fe90015ac10 2023-11-24T06:09:15.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.971+0000 7fe9084d8700 1 -- 172.21.15.91:0/4240924616 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9000cf540 con 0x7fe90015ac10 2023-11-24T06:09:15.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.972+0000 7fe9084d8700 1 -- 172.21.15.91:0/4240924616 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe900151b50 con 0x7fe90015ac10 2023-11-24T06:09:15.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.972+0000 7fe8f6ffd700 1 -- 172.21.15.91:0/4240924616 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fe8f80043e0 con 0x7fe90015ac10 2023-11-24T06:09:15.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.972+0000 7fe8f6ffd700 1 --2- 172.21.15.91:0/4240924616 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe8e406af60 0x7fe8e406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:15.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.973+0000 7fe8f6ffd700 1 -- 172.21.15.91:0/4240924616 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8f8022070 con 0x7fe90015ac10 2023-11-24T06:09:15.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.973+0000 7fe906274700 1 --2- 172.21.15.91:0/4240924616 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe8e406af60 0x7fe8e406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:15.978 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.973+0000 7fe906274700 1 --2- 172.21.15.91:0/4240924616 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe8e406af60 0x7fe8e406d420 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc006fd0 tx=0x7fe8fc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:15.979 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:15.977+0000 7fe8f6ffd700 1 -- 172.21.15.91:0/4240924616 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe900151b50 con 0x7fe90015ac10 2023-11-24T06:09:16.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.169+0000 7fe9084d8700 1 -- 172.21.15.91:0/4240924616 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe900151b20 con 0x7fe90015ac10 2023-11-24T06:09:16.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.169+0000 7fe8f6ffd700 1 -- 172.21.15.91:0/4240924616 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe8f8056d10 con 0x7fe90015ac10 2023-11-24T06:09:16.172 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 32 2023-11-24T06:09:16.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.171+0000 7fe9084d8700 1 -- 172.21.15.91:0/4240924616 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe8e406af60 msgr2=0x7fe8e406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:16.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.171+0000 7fe9084d8700 1 --2- 172.21.15.91:0/4240924616 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe8e406af60 0x7fe8e406d420 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc006fd0 tx=0x7fe8fc008040 comp rx=0 tx=0).stop 2023-11-24T06:09:16.175 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.171+0000 7fe9084d8700 1 -- 172.21.15.91:0/4240924616 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe90015ac10 msgr2=0x7fe900145800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:16.175 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.171+0000 7fe9084d8700 1 --2- 172.21.15.91:0/4240924616 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe90015ac10 0x7fe900145800 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8013000 tx=0x7fe8f8004ad0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.175 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.171+0000 7fe9084d8700 1 -- 172.21.15.91:0/4240924616 shutdown_connections 2023-11-24T06:09:16.175 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.171+0000 7fe9084d8700 1 --2- 172.21.15.91:0/4240924616 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fe8e406af60 0x7fe8e406d420 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.175 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.171+0000 7fe9084d8700 1 --2- 172.21.15.91:0/4240924616 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fe90015a440 0x7fe9000cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.175 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.171+0000 7fe9084d8700 1 --2- 172.21.15.91:0/4240924616 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fe90015ac10 0x7fe900145800 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.171+0000 7fe9084d8700 1 --2- 172.21.15.91:0/4240924616 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fe90015eb80 0x7fe9001452c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.171+0000 7fe9084d8700 1 -- 172.21.15.91:0/4240924616 >> 172.21.15.91:0/4240924616 conn(0x7fe9000b5070 msgr2=0x7fe900157d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:16.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.171+0000 7fe9084d8700 1 -- 172.21.15.91:0/4240924616 shutdown_connections 2023-11-24T06:09:16.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.171+0000 7fe9084d8700 1 -- 172.21.15.91:0/4240924616 wait complete. 2023-11-24T06:09:16.186 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:09:16.308 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.305+0000 7f7917a0c700 1 Processor -- start 2023-11-24T06:09:16.308 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.305+0000 7f7917a0c700 1 -- start start 2023-11-24T06:09:16.317 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7917a0c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f791013dd00 0x7f79100cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:16.317 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7917a0c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79100cf970 0x7f79100cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:16.318 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7917a0c700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791015f5b0 con 0x7f7910071b40 2023-11-24T06:09:16.318 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7917a0c700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f791013e0e0 con 0x7f791013dd00 2023-11-24T06:09:16.318 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7917a0c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791015e9a0 con 0x7f79100cf970 2023-11-24T06:09:16.318 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7915fa9700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f7910071b40 0x7f7910140490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:54982/0 (socket says 172.21.15.91:54982) 2023-11-24T06:09:16.318 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7915fa9700 1 -- 172.21.15.91:0/1481094894 learned_addr learned my addr 172.21.15.91:0/1481094894 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:16.318 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7914fa7700 1 --2- 172.21.15.91:0/1481094894 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79100cf970 0x7f79100cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:16.319 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f79157a8700 1 --2- 172.21.15.91:0/1481094894 >> v2:172.21.15.94:3300/0 conn(0x7f791013dd00 0x7f79100cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:16.319 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7907fff700 1 -- 172.21.15.91:0/1481094894 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805528820 0 0) 0x7f791015f5b0 con 0x7f7910071b40 2023-11-24T06:09:16.319 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7907fff700 1 -- 172.21.15.91:0/1481094894 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78fc0035b0 con 0x7f7910071b40 2023-11-24T06:09:16.319 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7907fff700 1 -- 172.21.15.91:0/1481094894 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 9591163 0 0) 0x7f78fc0035b0 con 0x7f7910071b40 2023-11-24T06:09:16.319 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7907fff700 1 -- 172.21.15.91:0/1481094894 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79100cf970 msgr2=0x7f79100cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:16.319 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7907fff700 1 --2- 172.21.15.91:0/1481094894 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79100cf970 0x7f79100cff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.320 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7907fff700 1 -- 172.21.15.91:0/1481094894 >> v2:172.21.15.94:3300/0 conn(0x7f791013dd00 msgr2=0x7f79100cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:16.320 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7907fff700 1 --2- 172.21.15.91:0/1481094894 >> v2:172.21.15.94:3300/0 conn(0x7f791013dd00 0x7f79100cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.320 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7907fff700 1 -- 172.21.15.91:0/1481094894 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791015ec80 con 0x7f7910071b40 2023-11-24T06:09:16.320 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7907fff700 1 -- 172.21.15.91:0/1481094894 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f79080019f0 con 0x7f7910071b40 2023-11-24T06:09:16.320 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7907fff700 1 -- 172.21.15.91:0/1481094894 >> v1:172.21.15.91:6789/0 conn(0x7f7910071b40 legacy=0x7f7910140490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:16.320 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7907fff700 1 --2- 172.21.15.91:0/1481094894 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79100cf970 0x7f78fc003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:16.321 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7907fff700 1 --2- 172.21.15.91:0/1481094894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78fc003f80 0x7f78fc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:16.321 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.306+0000 7f7907fff700 1 --2- 172.21.15.91:0/1481094894 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f78fc004d90 0x7f78fc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:16.321 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.307+0000 7f7907fff700 1 -- 172.21.15.91:0/1481094894 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791013e0e0 con 0x7f78fc004d90 2023-11-24T06:09:16.321 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.307+0000 7f7907fff700 1 -- 172.21.15.91:0/1481094894 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791015e9a0 con 0x7f78fc003f80 2023-11-24T06:09:16.321 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.307+0000 7f7907fff700 1 -- 172.21.15.91:0/1481094894 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78fc00ba10 con 0x7f79100cf970 2023-11-24T06:09:16.321 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.307+0000 7f7915fa9700 1 --2- 172.21.15.91:0/1481094894 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f78fc004d90 0x7f78fc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:16.322 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.307+0000 7f7914fa7700 1 --2- 172.21.15.91:0/1481094894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78fc003f80 0x7f78fc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:16.322 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.307+0000 7f79157a8700 1 --2- 172.21.15.91:0/1481094894 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79100cf970 0x7f78fc003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:16.322 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.307+0000 7f7915fa9700 1 -- 172.21.15.91:0/1481094894 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79100cf970 msgr2=0x7f78fc003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:16.322 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.307+0000 7f7915fa9700 1 --2- 172.21.15.91:0/1481094894 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79100cf970 0x7f78fc003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.322 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.307+0000 7f7915fa9700 1 -- 172.21.15.91:0/1481094894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78fc003f80 msgr2=0x7f78fc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:16.323 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.307+0000 7f7915fa9700 1 --2- 172.21.15.91:0/1481094894 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f78fc003f80 0x7f78fc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.323 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.307+0000 7f7915fa9700 1 -- 172.21.15.91:0/1481094894 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc0035b0 con 0x7f78fc004d90 2023-11-24T06:09:16.323 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.307+0000 7f79157a8700 1 --2- 172.21.15.91:0/1481094894 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f79100cf970 0x7f78fc003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:16.323 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.307+0000 7f7915fa9700 1 --2- 172.21.15.91:0/1481094894 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f78fc004d90 0x7f78fc005210 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f790800a8d0 tx=0x7f78fc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e6e993764621e99 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:16.323 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.307+0000 7f7907fff700 1 -- 172.21.15.91:0/1481094894 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790800a340 con 0x7f78fc004d90 2023-11-24T06:09:16.323 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.307+0000 7f7907fff700 1 -- 172.21.15.91:0/1481094894 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7908012600 con 0x7f78fc004d90 2023-11-24T06:09:16.324 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.307+0000 7f7907fff700 1 -- 172.21.15.91:0/1481094894 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79080129b0 con 0x7f78fc004d90 2023-11-24T06:09:16.324 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.308+0000 7f7917a0c700 1 -- 172.21.15.91:0/1481094894 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f78fc004d90 msgr2=0x7f78fc005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:16.324 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.308+0000 7f7917a0c700 1 --2- 172.21.15.91:0/1481094894 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f78fc004d90 0x7f78fc005210 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f790800a8d0 tx=0x7f78fc005880 comp rx=0 tx=0).stop 2023-11-24T06:09:16.324 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.308+0000 7f7915fa9700 1 -- 172.21.15.91:0/1481094894 reap_dead start 2023-11-24T06:09:16.324 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.308+0000 7f7917a0c700 1 -- 172.21.15.91:0/1481094894 shutdown_connections 2023-11-24T06:09:16.324 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.308+0000 7f7917a0c700 1 -- 172.21.15.91:0/1481094894 >> 172.21.15.91:0/1481094894 conn(0x7f79100d0b10 msgr2=0x7f79100d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:16.325 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.308+0000 7f7917a0c700 1 -- 172.21.15.91:0/1481094894 shutdown_connections 2023-11-24T06:09:16.325 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.308+0000 7f7917a0c700 1 -- 172.21.15.91:0/1481094894 wait complete. 2023-11-24T06:09:16.325 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.309+0000 7f7917a0c700 1 Processor -- start 2023-11-24T06:09:16.325 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.309+0000 7f7917a0c700 1 -- start start 2023-11-24T06:09:16.325 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.310+0000 7f7917a0c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f78fc004d90 0x7f79101567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:16.325 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.310+0000 7f7917a0c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f79101f5010 0x7f79101f5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:16.326 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.310+0000 7f7917a0c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79100b5a40 0x7f79100b5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:16.326 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.310+0000 7f7917a0c700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79100b6430 con 0x7f79101f5010 2023-11-24T06:09:16.326 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.310+0000 7f7917a0c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79101f5b80 con 0x7f79100b5a40 2023-11-24T06:09:16.326 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.310+0000 7f7917a0c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79101f5e60 con 0x7f78fc004d90 2023-11-24T06:09:16.326 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.310+0000 7f7914fa7700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f79101f5010 0x7f79101f5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:16.327 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.310+0000 7f7914fa7700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f79101f5010 0x7f79101f5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60058/0 (socket says 172.21.15.91:60058) 2023-11-24T06:09:16.327 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.310+0000 7f7914fa7700 1 -- 172.21.15.91:0/2680507899 learned_addr learned my addr 172.21.15.91:0/2680507899 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:16.327 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.311+0000 7f79157a8700 1 --2- 172.21.15.91:0/2680507899 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f78fc004d90 0x7f79101567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:16.327 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.311+0000 7f7915fa9700 1 --2- 172.21.15.91:0/2680507899 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79100b5a40 0x7f79100b5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:16.327 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.311+0000 7f7914fa7700 1 -- 172.21.15.91:0/2680507899 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f78fc004d90 msgr2=0x7f79101567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:16.327 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.311+0000 7f7914fa7700 1 --2- 172.21.15.91:0/2680507899 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f78fc004d90 0x7f79101567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.328 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.311+0000 7f7914fa7700 1 -- 172.21.15.91:0/2680507899 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79100b5a40 msgr2=0x7f79100b5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:16.328 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.311+0000 7f7914fa7700 1 --2- 172.21.15.91:0/2680507899 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79100b5a40 0x7f79100b5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.328 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.311+0000 7f7914fa7700 1 -- 172.21.15.91:0/2680507899 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79100c8fe0 con 0x7f79101f5010 2023-11-24T06:09:16.328 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.311+0000 7f7914fa7700 1 --2- 172.21.15.91:0/2680507899 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f79101f5010 0x7f79101f5490 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7900011940 tx=0x7f7900011d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:16.328 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.311+0000 7f7905ffb700 1 -- 172.21.15.91:0/2680507899 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7900004240 con 0x7f79101f5010 2023-11-24T06:09:16.329 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.311+0000 7f7905ffb700 1 -- 172.21.15.91:0/2680507899 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79000043a0 con 0x7f79101f5010 2023-11-24T06:09:16.329 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.311+0000 7f7917a0c700 1 -- 172.21.15.91:0/2680507899 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79100c92c0 con 0x7f79101f5010 2023-11-24T06:09:16.329 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.311+0000 7f7905ffb700 1 -- 172.21.15.91:0/2680507899 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790000f950 con 0x7f79101f5010 2023-11-24T06:09:16.329 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.311+0000 7f7917a0c700 1 -- 172.21.15.91:0/2680507899 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7910156ce0 con 0x7f79101f5010 2023-11-24T06:09:16.329 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.312+0000 7f7917a0c700 1 -- 172.21.15.91:0/2680507899 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7910075480 con 0x7f79101f5010 2023-11-24T06:09:16.329 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.312+0000 7f7905ffb700 1 -- 172.21.15.91:0/2680507899 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f790000fab0 con 0x7f79101f5010 2023-11-24T06:09:16.330 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.313+0000 7f7905ffb700 1 --2- 172.21.15.91:0/2680507899 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f78f406af60 0x7f78f406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:16.330 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.313+0000 7f7905ffb700 1 -- 172.21.15.91:0/2680507899 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7900089ac0 con 0x7f79101f5010 2023-11-24T06:09:16.330 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.313+0000 7f79157a8700 1 --2- 172.21.15.91:0/2680507899 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f78f406af60 0x7f78f406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:16.330 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.313+0000 7f79157a8700 1 --2- 172.21.15.91:0/2680507899 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f78f406af60 0x7f78f406d420 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f790c009ab0 tx=0x7f790c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:16.331 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.317+0000 7f7905ffb700 1 -- 172.21.15.91:0/2680507899 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7900054c10 con 0x7f79101f5010 2023-11-24T06:09:16.512 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.509+0000 7f7917a0c700 1 -- 172.21.15.91:0/2680507899 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7910072ca0 con 0x7f79101f5010 2023-11-24T06:09:16.512 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.509+0000 7f7905ffb700 1 -- 172.21.15.91:0/2680507899 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7f79000230a0 con 0x7f79101f5010 2023-11-24T06:09:16.513 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 64 2023-11-24T06:09:16.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.512+0000 7f7917a0c700 1 -- 172.21.15.91:0/2680507899 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f78f406af60 msgr2=0x7f78f406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:16.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.512+0000 7f7917a0c700 1 --2- 172.21.15.91:0/2680507899 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f78f406af60 0x7f78f406d420 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f790c009ab0 tx=0x7f790c008040 comp rx=0 tx=0).stop 2023-11-24T06:09:16.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.512+0000 7f7917a0c700 1 -- 172.21.15.91:0/2680507899 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f79101f5010 msgr2=0x7f79101f5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:16.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.512+0000 7f7917a0c700 1 --2- 172.21.15.91:0/2680507899 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f79101f5010 0x7f79101f5490 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7900011940 tx=0x7f7900011d00 comp rx=0 tx=0).stop 2023-11-24T06:09:16.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.512+0000 7f7917a0c700 1 -- 172.21.15.91:0/2680507899 shutdown_connections 2023-11-24T06:09:16.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.512+0000 7f7917a0c700 1 --2- 172.21.15.91:0/2680507899 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f78f406af60 0x7f78f406d420 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.512+0000 7f7917a0c700 1 --2- 172.21.15.91:0/2680507899 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f79100b5a40 0x7f79100b5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.512+0000 7f7917a0c700 1 --2- 172.21.15.91:0/2680507899 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f78fc004d90 0x7f79101567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.512+0000 7f7917a0c700 1 --2- 172.21.15.91:0/2680507899 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f79101f5010 0x7f79101f5490 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.512+0000 7f7917a0c700 1 -- 172.21.15.91:0/2680507899 >> 172.21.15.91:0/2680507899 conn(0x7f79100d0b10 msgr2=0x7f7910158650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:16.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.512+0000 7f7917a0c700 1 -- 172.21.15.91:0/2680507899 shutdown_connections 2023-11-24T06:09:16.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.512+0000 7f7917a0c700 1 -- 172.21.15.91:0/2680507899 wait complete. 2023-11-24T06:09:16.527 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-11-24T06:09:16.647 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.644+0000 7f4c8cb07700 1 Processor -- start 2023-11-24T06:09:16.647 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.644+0000 7f4c8cb07700 1 -- start start 2023-11-24T06:09:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.645+0000 7f4c8cb07700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4c8815bde0 0x7f4c8815c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.645+0000 7f4c8cb07700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4c880cf2b0 0x7f4c880d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.645+0000 7f4c8cb07700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c880d0a90 con 0x7f4c880ce5c0 2023-11-24T06:09:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.645+0000 7f4c8cb07700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c8806ee10 con 0x7f4c8815bde0 2023-11-24T06:09:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.645+0000 7f4c8cb07700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8806f0f0 con 0x7f4c880cf2b0 2023-11-24T06:09:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.646+0000 7f4c8659c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4c8815bde0 0x7f4c8815c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.646+0000 7f4c8659c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4c8815bde0 0x7f4c8815c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60064/0 (socket says 172.21.15.91:60064) 2023-11-24T06:09:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.646+0000 7f4c8659c700 1 -- 172.21.15.91:0/3645351522 learned_addr learned my addr 172.21.15.91:0/3645351522 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.646+0000 7f4c85d9b700 1 --2- 172.21.15.91:0/3645351522 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4c880cf2b0 0x7f4c880d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:16.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.646+0000 7f4c8659c700 1 -- 172.21.15.91:0/3645351522 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4c880cf2b0 msgr2=0x7f4c880d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.646+0000 7f4c8659c700 1 --2- 172.21.15.91:0/3645351522 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4c880cf2b0 0x7f4c880d01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.646+0000 7f4c8659c700 1 -- 172.21.15.91:0/3645351522 >> v1:172.21.15.94:6789/0 conn(0x7f4c880ce5c0 legacy=0x7f4c880cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.646+0000 7f4c8659c700 1 -- 172.21.15.91:0/3645351522 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8806f3d0 con 0x7f4c8815bde0 2023-11-24T06:09:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.646+0000 7f4c8659c700 1 --2- 172.21.15.91:0/3645351522 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c8815bde0 0x7f4c8815c1c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4c780098f0 tx=0x7f4c78009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a2809620c73748 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.646+0000 7f4c8559a700 1 -- 172.21.15.91:0/3645351522 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c78004230 con 0x7f4c8815bde0 2023-11-24T06:09:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.646+0000 7f4c8559a700 1 -- 172.21.15.91:0/3645351522 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c78004390 con 0x7f4c8815bde0 2023-11-24T06:09:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.646+0000 7f4c8559a700 1 -- 172.21.15.91:0/3645351522 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c780036a0 con 0x7f4c8815bde0 2023-11-24T06:09:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.647+0000 7f4c8cb07700 1 -- 172.21.15.91:0/3645351522 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c8815bde0 msgr2=0x7f4c8815c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.647+0000 7f4c8cb07700 1 --2- 172.21.15.91:0/3645351522 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c8815bde0 0x7f4c8815c1c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4c780098f0 tx=0x7f4c78009c00 comp rx=0 tx=0).stop 2023-11-24T06:09:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.647+0000 7f4c8cb07700 1 -- 172.21.15.91:0/3645351522 shutdown_connections 2023-11-24T06:09:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.647+0000 7f4c8cb07700 1 --2- 172.21.15.91:0/3645351522 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c8815bde0 0x7f4c8815c1c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.647+0000 7f4c8cb07700 1 --2- 172.21.15.91:0/3645351522 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4c880cf2b0 0x7f4c880d01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.651 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.647+0000 7f4c8cb07700 1 -- 172.21.15.91:0/3645351522 >> 172.21.15.91:0/3645351522 conn(0x7f4c880b3720 msgr2=0x7f4c880b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:16.652 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.647+0000 7f4c8cb07700 1 -- 172.21.15.91:0/3645351522 shutdown_connections 2023-11-24T06:09:16.652 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.647+0000 7f4c8cb07700 1 -- 172.21.15.91:0/3645351522 wait complete. 2023-11-24T06:09:16.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.648+0000 7f4c8cb07700 1 Processor -- start 2023-11-24T06:09:16.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.648+0000 7f4c8cb07700 1 -- start start 2023-11-24T06:09:16.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.649+0000 7f4c8cb07700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c880ce5c0 0x7f4c880b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:16.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.649+0000 7f4c8cb07700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c880cf2b0 0x7f4c8815e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:16.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.649+0000 7f4c8cb07700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4c8815bde0 0x7f4c88159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:16.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.649+0000 7f4c8cb07700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c880b0b00 con 0x7f4c880cf2b0 2023-11-24T06:09:16.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.649+0000 7f4c8cb07700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8809f9d0 con 0x7f4c880ce5c0 2023-11-24T06:09:16.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.649+0000 7f4c8cb07700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c880b5cd0 con 0x7f4c8815bde0 2023-11-24T06:09:16.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.649+0000 7f4c85d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c880cf2b0 0x7f4c8815e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:16.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.649+0000 7f4c85d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c880cf2b0 0x7f4c8815e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60072/0 (socket says 172.21.15.91:60072) 2023-11-24T06:09:16.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.649+0000 7f4c85d9b700 1 -- 172.21.15.91:0/2520898198 learned_addr learned my addr 172.21.15.91:0/2520898198 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:16.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.649+0000 7f4c8659c700 1 --2- 172.21.15.91:0/2520898198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c880ce5c0 0x7f4c880b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:16.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.649+0000 7f4c86d9d700 1 --2- 172.21.15.91:0/2520898198 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4c8815bde0 0x7f4c88159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:16.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.649+0000 7f4c85d9b700 1 -- 172.21.15.91:0/2520898198 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4c8815bde0 msgr2=0x7f4c88159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:16.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.649+0000 7f4c85d9b700 1 --2- 172.21.15.91:0/2520898198 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4c8815bde0 0x7f4c88159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.649+0000 7f4c85d9b700 1 -- 172.21.15.91:0/2520898198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c880ce5c0 msgr2=0x7f4c880b05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:16.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.649+0000 7f4c85d9b700 1 --2- 172.21.15.91:0/2520898198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c880ce5c0 0x7f4c880b05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.649+0000 7f4c85d9b700 1 -- 172.21.15.91:0/2520898198 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c78009550 con 0x7f4c880cf2b0 2023-11-24T06:09:16.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.650+0000 7f4c85d9b700 1 --2- 172.21.15.91:0/2520898198 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c880cf2b0 0x7f4c8815e440 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c00c7f0 tx=0x7f4c7c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:16.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.650+0000 7f4c76ffd700 1 -- 172.21.15.91:0/2520898198 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c004230 con 0x7f4c880cf2b0 2023-11-24T06:09:16.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.650+0000 7f4c76ffd700 1 -- 172.21.15.91:0/2520898198 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7c004390 con 0x7f4c880cf2b0 2023-11-24T06:09:16.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.650+0000 7f4c8cb07700 1 -- 172.21.15.91:0/2520898198 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c880b5fb0 con 0x7f4c880cf2b0 2023-11-24T06:09:16.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.650+0000 7f4c76ffd700 1 -- 172.21.15.91:0/2520898198 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c0036a0 con 0x7f4c880cf2b0 2023-11-24T06:09:16.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.650+0000 7f4c8cb07700 1 -- 172.21.15.91:0/2520898198 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c880b6570 con 0x7f4c880cf2b0 2023-11-24T06:09:16.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.651+0000 7f4c8cb07700 1 -- 172.21.15.91:0/2520898198 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c88068670 con 0x7f4c880cf2b0 2023-11-24T06:09:16.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.651+0000 7f4c76ffd700 1 -- 172.21.15.91:0/2520898198 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f4c7c003800 con 0x7f4c880cf2b0 2023-11-24T06:09:16.657 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.652+0000 7f4c76ffd700 1 --2- 172.21.15.91:0/2520898198 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c6806af90 0x7f4c6806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:16.657 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.652+0000 7f4c76ffd700 1 -- 172.21.15.91:0/2520898198 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c7c089370 con 0x7f4c880cf2b0 2023-11-24T06:09:16.657 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.652+0000 7f4c8659c700 1 --2- 172.21.15.91:0/2520898198 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c6806af90 0x7f4c6806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:16.657 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.652+0000 7f4c8659c700 1 --2- 172.21.15.91:0/2520898198 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c6806af90 0x7f4c6806d450 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009130 tx=0x7f4c78008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:16.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.656+0000 7f4c76ffd700 1 -- 172.21.15.91:0/2520898198 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c7c054320 con 0x7f4c880cf2b0 2023-11-24T06:09:16.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.862+0000 7f4c8cb07700 1 -- 172.21.15.91:0/2520898198 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4c880d0140 con 0x7f4c880cf2b0 2023-11-24T06:09:16.866 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.863+0000 7f4c76ffd700 1 -- 172.21.15.91:0/2520898198 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4c7c057940 con 0x7f4c880cf2b0 2023-11-24T06:09:16.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.865+0000 7f4c8cb07700 1 -- 172.21.15.91:0/2520898198 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c6806af90 msgr2=0x7f4c6806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:16.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.865+0000 7f4c8cb07700 1 --2- 172.21.15.91:0/2520898198 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c6806af90 0x7f4c6806d450 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009130 tx=0x7f4c78008030 comp rx=0 tx=0).stop 2023-11-24T06:09:16.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.865+0000 7f4c8cb07700 1 -- 172.21.15.91:0/2520898198 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c880cf2b0 msgr2=0x7f4c8815e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:16.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.865+0000 7f4c8cb07700 1 --2- 172.21.15.91:0/2520898198 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c880cf2b0 0x7f4c8815e440 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c00c7f0 tx=0x7f4c7c00cb00 comp rx=0 tx=0).stop 2023-11-24T06:09:16.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.865+0000 7f4c8cb07700 1 -- 172.21.15.91:0/2520898198 shutdown_connections 2023-11-24T06:09:16.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.865+0000 7f4c8cb07700 1 --2- 172.21.15.91:0/2520898198 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4c6806af90 0x7f4c6806d450 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.865+0000 7f4c8cb07700 1 --2- 172.21.15.91:0/2520898198 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4c880ce5c0 0x7f4c880b05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.865+0000 7f4c8cb07700 1 --2- 172.21.15.91:0/2520898198 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4c8815bde0 0x7f4c88159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.865+0000 7f4c8cb07700 1 --2- 172.21.15.91:0/2520898198 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4c880cf2b0 0x7f4c8815e440 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:16.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.865+0000 7f4c8cb07700 1 -- 172.21.15.91:0/2520898198 >> 172.21.15.91:0/2520898198 conn(0x7f4c880b3720 msgr2=0x7f4c880aef60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:16.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.865+0000 7f4c8cb07700 1 -- 172.21.15.91:0/2520898198 shutdown_connections 2023-11-24T06:09:16.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:16.865+0000 7f4c8cb07700 1 -- 172.21.15.91:0/2520898198 wait complete. 2023-11-24T06:09:16.869 INFO:teuthology.orchestra.run.smithi091.stderr:listed 0 entries 2023-11-24T06:09:16.880 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-11-24T06:09:16.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:16.880 DEBUG:teuthology.orchestra.run.smithi094:> sudo umount /home/ubuntu/cephtest/mnt.1 2023-11-24T06:09:16.936 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi094.front.sepia.ceph.com 2023-11-24T06:09:16.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:16.936 DEBUG:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-11-24T06:09:16.956 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:09:16.956 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:09:16.973 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13970 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-24T06:09:16.973 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:09:16.973 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:09:17.015 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16729 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-24T06:09:17.015 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:09:17.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.131+0000 7f51e19be700 1 Processor -- start 2023-11-24T06:09:17.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.131+0000 7f51e19be700 1 -- start start 2023-11-24T06:09:17.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.132+0000 7f51e19be700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f51dc1459a0 0x7f51dc0cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.132+0000 7f51e19be700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51dc0cf5f0 con 0x7f51dc142810 2023-11-24T06:09:17.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.132+0000 7f51e19be700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51dc06ea80 con 0x7f51dc06b6e0 2023-11-24T06:09:17.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.132+0000 7f51e19be700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc06ed60 con 0x7f51dc1459a0 2023-11-24T06:09:17.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.132+0000 7f51daffd700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f51dc142810 0x7f51dc06afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55012/0 (socket says 172.21.15.91:55012) 2023-11-24T06:09:17.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.132+0000 7f51daffd700 1 -- 172.21.15.91:0/1297899386 learned_addr learned my addr 172.21.15.91:0/1297899386 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:17.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51d9ffb700 1 -- 172.21.15.91:0/1297899386 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383497625 0 0) 0x7f51dc0cf5f0 con 0x7f51dc142810 2023-11-24T06:09:17.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51d9ffb700 1 -- 172.21.15.91:0/1297899386 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51c40035b0 con 0x7f51dc142810 2023-11-24T06:09:17.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51da7fc700 1 --2- 172.21.15.91:0/1297899386 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f51dc1459a0 0x7f51dc0cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51d9ffb700 1 -- 172.21.15.91:0/1297899386 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 116318062 0 0) 0x7f51c40035b0 con 0x7f51dc142810 2023-11-24T06:09:17.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51d9ffb700 1 -- 172.21.15.91:0/1297899386 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f51dc1459a0 msgr2=0x7f51dc0cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:17.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51d9ffb700 1 --2- 172.21.15.91:0/1297899386 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f51dc1459a0 0x7f51dc0cfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51d9ffb700 1 -- 172.21.15.91:0/1297899386 >> v1:172.21.15.94:6789/0 conn(0x7f51dc06b6e0 legacy=0x7f51dc0cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51d9ffb700 1 -- 172.21.15.91:0/1297899386 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51dc06f040 con 0x7f51dc142810 2023-11-24T06:09:17.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51d9ffb700 1 -- 172.21.15.91:0/1297899386 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7f51d0001670 con 0x7f51dc142810 2023-11-24T06:09:17.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51d9ffb700 1 -- 172.21.15.91:0/1297899386 >> v1:172.21.15.91:6789/0 conn(0x7f51dc142810 legacy=0x7f51dc06afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51d9ffb700 1 --2- 172.21.15.91:0/1297899386 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f51c4003a30 0x7f51c4003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51d9ffb700 1 --2- 172.21.15.91:0/1297899386 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f51dc1459a0 0x7f51c4004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51d9ffb700 1 --2- 172.21.15.91:0/1297899386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51c4004d90 0x7f51c4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51d9ffb700 1 -- 172.21.15.91:0/1297899386 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc06ed60 con 0x7f51c4003a30 2023-11-24T06:09:17.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51d9ffb700 1 -- 172.21.15.91:0/1297899386 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c400bbf0 con 0x7f51c4004d90 2023-11-24T06:09:17.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51d9ffb700 1 -- 172.21.15.91:0/1297899386 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c400bdb0 con 0x7f51dc1459a0 2023-11-24T06:09:17.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51daffd700 1 --2- 172.21.15.91:0/1297899386 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f51c4003a30 0x7f51c4003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51da7fc700 1 --2- 172.21.15.91:0/1297899386 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f51dc1459a0 0x7f51c4004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51db7fe700 1 --2- 172.21.15.91:0/1297899386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51c4004d90 0x7f51c4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51daffd700 1 -- 172.21.15.91:0/1297899386 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f51dc1459a0 msgr2=0x7f51c4004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:17.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51daffd700 1 --2- 172.21.15.91:0/1297899386 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f51dc1459a0 0x7f51c4004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51daffd700 1 -- 172.21.15.91:0/1297899386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51c4004d90 msgr2=0x7f51c4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:17.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51daffd700 1 --2- 172.21.15.91:0/1297899386 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51c4004d90 0x7f51c4005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51daffd700 1 -- 172.21.15.91:0/1297899386 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c40035b0 con 0x7f51c4003a30 2023-11-24T06:09:17.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.133+0000 7f51daffd700 1 --2- 172.21.15.91:0/1297899386 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f51c4003a30 0x7f51c4003e50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f51d000c240 tx=0x7f51c4005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13c43f95a34437a4 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:17.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.134+0000 7f51d9ffb700 1 -- 172.21.15.91:0/1297899386 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0007c10 con 0x7f51c4003a30 2023-11-24T06:09:17.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.134+0000 7f51d9ffb700 1 -- 172.21.15.91:0/1297899386 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d000ce00 con 0x7f51c4003a30 2023-11-24T06:09:17.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.134+0000 7f51d9ffb700 1 -- 172.21.15.91:0/1297899386 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0005ba0 con 0x7f51c4003a30 2023-11-24T06:09:17.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.134+0000 7f51e19be700 1 -- 172.21.15.91:0/1297899386 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f51c4003a30 msgr2=0x7f51c4003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.134+0000 7f51e19be700 1 --2- 172.21.15.91:0/1297899386 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f51c4003a30 0x7f51c4003e50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f51d000c240 tx=0x7f51c4005880 comp rx=0 tx=0).stop 2023-11-24T06:09:17.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.134+0000 7f51db7fe700 1 -- 172.21.15.91:0/1297899386 reap_dead start 2023-11-24T06:09:17.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.134+0000 7f51e19be700 1 -- 172.21.15.91:0/1297899386 shutdown_connections 2023-11-24T06:09:17.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.134+0000 7f51e19be700 1 -- 172.21.15.91:0/1297899386 >> 172.21.15.91:0/1297899386 conn(0x7f51dc0b2740 msgr2=0x7f51dc0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:17.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.134+0000 7f51e19be700 1 -- 172.21.15.91:0/1297899386 shutdown_connections 2023-11-24T06:09:17.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.134+0000 7f51e19be700 1 -- 172.21.15.91:0/1297899386 wait complete. 2023-11-24T06:09:17.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.135+0000 7f51e19be700 1 Processor -- start 2023-11-24T06:09:17.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.136+0000 7f51e19be700 1 -- start start 2023-11-24T06:09:17.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.136+0000 7f51e19be700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f51c4003a30 0x7f51dc13d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.136+0000 7f51e19be700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f51dc0af310 0x7f51dc0b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.136+0000 7f51e19be700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51dc0af790 0x7f51dc0b13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.136+0000 7f51e19be700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc14d9d0 con 0x7f51c4003a30 2023-11-24T06:09:17.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.136+0000 7f51e19be700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc13db00 con 0x7f51dc0af790 2023-11-24T06:09:17.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.136+0000 7f51e19be700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc14d630 con 0x7f51dc0af310 2023-11-24T06:09:17.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.136+0000 7f51daffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f51c4003a30 0x7f51dc13d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.136+0000 7f51daffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f51c4003a30 0x7f51dc13d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60090/0 (socket says 172.21.15.91:60090) 2023-11-24T06:09:17.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.136+0000 7f51daffd700 1 -- 172.21.15.91:0/2916826847 learned_addr learned my addr 172.21.15.91:0/2916826847 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:17.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.137+0000 7f51da7fc700 1 --2- 172.21.15.91:0/2916826847 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f51dc0af310 0x7f51dc0b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.137+0000 7f51daffd700 1 -- 172.21.15.91:0/2916826847 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f51dc0af310 msgr2=0x7f51dc0b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.137+0000 7f51daffd700 1 --2- 172.21.15.91:0/2916826847 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f51dc0af310 0x7f51dc0b0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.137+0000 7f51db7fe700 1 --2- 172.21.15.91:0/2916826847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51dc0af790 0x7f51dc0b13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.137+0000 7f51daffd700 1 -- 172.21.15.91:0/2916826847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51dc0af790 msgr2=0x7f51dc0b13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.137+0000 7f51daffd700 1 --2- 172.21.15.91:0/2916826847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51dc0af790 0x7f51dc0b13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.137+0000 7f51daffd700 1 -- 172.21.15.91:0/2916826847 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51dc0afcf0 con 0x7f51c4003a30 2023-11-24T06:09:17.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.137+0000 7f51db7fe700 1 --2- 172.21.15.91:0/2916826847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51dc0af790 0x7f51dc0b13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:17.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.137+0000 7f51daffd700 1 --2- 172.21.15.91:0/2916826847 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f51c4003a30 0x7f51dc13d5c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f51c400bbc0 tx=0x7f51d0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:17.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.137+0000 7f51c3fff700 1 -- 172.21.15.91:0/2916826847 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0001630 con 0x7f51c4003a30 2023-11-24T06:09:17.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.137+0000 7f51c3fff700 1 -- 172.21.15.91:0/2916826847 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d00070d0 con 0x7f51c4003a30 2023-11-24T06:09:17.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.137+0000 7f51c3fff700 1 -- 172.21.15.91:0/2916826847 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0007480 con 0x7f51c4003a30 2023-11-24T06:09:17.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.137+0000 7f51e19be700 1 -- 172.21.15.91:0/2916826847 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51dc0affd0 con 0x7f51c4003a30 2023-11-24T06:09:17.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.137+0000 7f51e19be700 1 -- 172.21.15.91:0/2916826847 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51dc153780 con 0x7f51c4003a30 2023-11-24T06:09:17.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.138+0000 7f51e19be700 1 -- 172.21.15.91:0/2916826847 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51dc068670 con 0x7f51c4003a30 2023-11-24T06:09:17.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.138+0000 7f51c3fff700 1 -- 172.21.15.91:0/2916826847 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f51d0019030 con 0x7f51c4003a30 2023-11-24T06:09:17.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.139+0000 7f51c3fff700 1 --2- 172.21.15.91:0/2916826847 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f51c806af60 0x7f51c806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.139+0000 7f51c3fff700 1 -- 172.21.15.91:0/2916826847 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51d008a670 con 0x7f51c4003a30 2023-11-24T06:09:17.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.139+0000 7f51da7fc700 1 --2- 172.21.15.91:0/2916826847 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f51c806af60 0x7f51c806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.147 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.139+0000 7f51da7fc700 1 --2- 172.21.15.91:0/2916826847 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f51c806af60 0x7f51c806d420 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f51cc009790 tx=0x7f51cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:17.147 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.143+0000 7f51c3fff700 1 -- 172.21.15.91:0/2916826847 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51d0055740 con 0x7f51c4003a30 2023-11-24T06:09:17.378 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.375+0000 7f51e19be700 1 -- 172.21.15.91:0/2916826847 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51dc0b2bc0 con 0x7f51c4003a30 2023-11-24T06:09:17.379 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.376+0000 7f51c3fff700 1 -- 172.21.15.91:0/2916826847 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4907 (secure 0 0 0) 0x7f51d001ae60 con 0x7f51c4003a30 2023-11-24T06:09:17.379 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:17.379 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4706,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1685174039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1685174039},{"type":"v1","addr":"172.21.15.94:6834","nonce":1685174039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4709,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1508344631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1508344631},{"type":"v1","addr":"172.21.15.94:6835","nonce":1508344631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5064,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6836/28114058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6834","nonce":28114058},{"type":"v1","addr":"172.21.15.91:6836","nonce":28114058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:08:58.437951+0000","modified":"2023-11-24T06:09:01.476634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.91:6837/1301930708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":1301930708},{"type":"v1","addr":"172.21.15.91:6837","nonce":1301930708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T06:09:17.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.378+0000 7f51e19be700 1 -- 172.21.15.91:0/2916826847 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f51c806af60 msgr2=0x7f51c806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.378+0000 7f51e19be700 1 --2- 172.21.15.91:0/2916826847 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f51c806af60 0x7f51c806d420 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f51cc009790 tx=0x7f51cc008040 comp rx=0 tx=0).stop 2023-11-24T06:09:17.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.378+0000 7f51e19be700 1 -- 172.21.15.91:0/2916826847 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f51c4003a30 msgr2=0x7f51dc13d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.378+0000 7f51e19be700 1 --2- 172.21.15.91:0/2916826847 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f51c4003a30 0x7f51dc13d5c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f51c400bbc0 tx=0x7f51d0001a00 comp rx=0 tx=0).stop 2023-11-24T06:09:17.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.378+0000 7f51e19be700 1 -- 172.21.15.91:0/2916826847 shutdown_connections 2023-11-24T06:09:17.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.378+0000 7f51e19be700 1 --2- 172.21.15.91:0/2916826847 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f51c806af60 0x7f51c806d420 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.381 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.378+0000 7f51e19be700 1 --2- 172.21.15.91:0/2916826847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f51dc0af790 0x7f51dc0b13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.382 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.378+0000 7f51e19be700 1 --2- 172.21.15.91:0/2916826847 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f51c4003a30 0x7f51dc13d5c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.382 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.378+0000 7f51e19be700 1 --2- 172.21.15.91:0/2916826847 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f51dc0af310 0x7f51dc0b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.382 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.378+0000 7f51e19be700 1 -- 172.21.15.91:0/2916826847 >> 172.21.15.91:0/2916826847 conn(0x7f51dc0b2740 msgr2=0x7f51dc13f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:17.382 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.378+0000 7f51e19be700 1 -- 172.21.15.91:0/2916826847 shutdown_connections 2023-11-24T06:09:17.382 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.378+0000 7f51e19be700 1 -- 172.21.15.91:0/2916826847 wait complete. 2023-11-24T06:09:17.382 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 17 2023-11-24T06:09:17.393 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:09:17.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.512+0000 7fb9e4e1e700 1 Processor -- start 2023-11-24T06:09:17.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.512+0000 7fb9e4e1e700 1 -- start start 2023-11-24T06:09:17.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.513+0000 7fb9e4e1e700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fb9e006fc50 0x7fb9e006eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.513+0000 7fb9e4e1e700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e00d1230 con 0x7fb9e015e1c0 2023-11-24T06:09:17.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.513+0000 7fb9e4e1e700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e00d0c70 con 0x7fb9e00b0e90 2023-11-24T06:09:17.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.513+0000 7fb9e4e1e700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9e00d0f50 con 0x7fb9e006fc50 2023-11-24T06:09:17.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9de59c700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fb9e015e1c0 0x7fb9e006f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55032/0 (socket says 172.21.15.91:55032) 2023-11-24T06:09:17.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9de59c700 1 -- 172.21.15.91:0/1052219734 learned_addr learned my addr 172.21.15.91:0/1052219734 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:17.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9ddd9b700 1 --2- 172.21.15.91:0/1052219734 >> v2:172.21.15.94:3300/0 conn(0x7fb9e006fc50 0x7fb9e006eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9dd59a700 1 -- 172.21.15.91:0/1052219734 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149628998 0 0) 0x7fb9e00d1230 con 0x7fb9e015e1c0 2023-11-24T06:09:17.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9dd59a700 1 -- 172.21.15.91:0/1052219734 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c80035b0 con 0x7fb9e015e1c0 2023-11-24T06:09:17.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9dd59a700 1 -- 172.21.15.91:0/1052219734 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2827015302 0 0) 0x7fb9c80035b0 con 0x7fb9e015e1c0 2023-11-24T06:09:17.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9dd59a700 1 -- 172.21.15.91:0/1052219734 >> v2:172.21.15.94:3300/0 conn(0x7fb9e006fc50 msgr2=0x7fb9e006eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:17.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9dd59a700 1 --2- 172.21.15.91:0/1052219734 >> v2:172.21.15.94:3300/0 conn(0x7fb9e006fc50 0x7fb9e006eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9dd59a700 1 -- 172.21.15.91:0/1052219734 >> v1:172.21.15.94:6789/0 conn(0x7fb9e00b0e90 legacy=0x7fb9e006ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9dd59a700 1 -- 172.21.15.91:0/1052219734 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e006c210 con 0x7fb9e015e1c0 2023-11-24T06:09:17.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9dd59a700 1 -- 172.21.15.91:0/1052219734 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7fb9d4001670 con 0x7fb9e015e1c0 2023-11-24T06:09:17.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9dd59a700 1 -- 172.21.15.91:0/1052219734 >> v1:172.21.15.91:6789/0 conn(0x7fb9e015e1c0 legacy=0x7fb9e006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9dd59a700 1 --2- 172.21.15.91:0/1052219734 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb9c8003a00 0x7fb9c8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9dd59a700 1 --2- 172.21.15.91:0/1052219734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9c8004850 0x7fb9c8004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.521 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9dd59a700 1 --2- 172.21.15.91:0/1052219734 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb9c8005210 0x7fb9c800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.521 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9dd59a700 1 -- 172.21.15.91:0/1052219734 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e00d0f50 con 0x7fb9c8003a00 2023-11-24T06:09:17.521 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9dd59a700 1 -- 172.21.15.91:0/1052219734 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c800c1b0 con 0x7fb9c8004850 2023-11-24T06:09:17.521 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9dd59a700 1 -- 172.21.15.91:0/1052219734 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c800c370 con 0x7fb9c8005210 2023-11-24T06:09:17.521 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9de59c700 1 --2- 172.21.15.91:0/1052219734 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb9c8003a00 0x7fb9c8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.521 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9ddd9b700 1 --2- 172.21.15.91:0/1052219734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9c8004850 0x7fb9c8004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.521 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.514+0000 7fb9ded9d700 1 --2- 172.21.15.91:0/1052219734 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb9c8005210 0x7fb9c800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.521 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.515+0000 7fb9de59c700 1 -- 172.21.15.91:0/1052219734 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb9c8005210 msgr2=0x7fb9c800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:17.521 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.515+0000 7fb9de59c700 1 --2- 172.21.15.91:0/1052219734 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb9c8005210 0x7fb9c800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.521 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.515+0000 7fb9de59c700 1 -- 172.21.15.91:0/1052219734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9c8004850 msgr2=0x7fb9c8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:17.521 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.515+0000 7fb9de59c700 1 --2- 172.21.15.91:0/1052219734 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9c8004850 0x7fb9c8004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.521 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.515+0000 7fb9ded9d700 1 -- 172.21.15.91:0/1052219734 reap_dead start 2023-11-24T06:09:17.521 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.515+0000 7fb9de59c700 1 -- 172.21.15.91:0/1052219734 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c80035b0 con 0x7fb9c8003a00 2023-11-24T06:09:17.522 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.515+0000 7fb9de59c700 1 --2- 172.21.15.91:0/1052219734 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb9c8003a00 0x7fb9c8003e20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb9d400c240 tx=0x7fb9c8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c05bec3e247959b9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:17.522 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.515+0000 7fb9dd59a700 1 -- 172.21.15.91:0/1052219734 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d4007c10 con 0x7fb9c8003a00 2023-11-24T06:09:17.522 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.515+0000 7fb9dd59a700 1 -- 172.21.15.91:0/1052219734 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d400cda0 con 0x7fb9c8003a00 2023-11-24T06:09:17.522 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.515+0000 7fb9dd59a700 1 -- 172.21.15.91:0/1052219734 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d4005ba0 con 0x7fb9c8003a00 2023-11-24T06:09:17.522 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.515+0000 7fb9e4e1e700 1 -- 172.21.15.91:0/1052219734 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb9c8003a00 msgr2=0x7fb9c8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.522 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.515+0000 7fb9e4e1e700 1 --2- 172.21.15.91:0/1052219734 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb9c8003a00 0x7fb9c8003e20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb9d400c240 tx=0x7fb9c8005880 comp rx=0 tx=0).stop 2023-11-24T06:09:17.522 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.515+0000 7fb9e4e1e700 1 -- 172.21.15.91:0/1052219734 shutdown_connections 2023-11-24T06:09:17.522 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.515+0000 7fb9e4e1e700 1 --2- 172.21.15.91:0/1052219734 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb9c8003a00 0x7fb9c8003e20 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.522 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.515+0000 7fb9e4e1e700 1 -- 172.21.15.91:0/1052219734 >> 172.21.15.91:0/1052219734 conn(0x7fb9e00b3720 msgr2=0x7fb9e00b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:17.522 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.516+0000 7fb9e4e1e700 1 -- 172.21.15.91:0/1052219734 shutdown_connections 2023-11-24T06:09:17.522 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.516+0000 7fb9e4e1e700 1 -- 172.21.15.91:0/1052219734 wait complete. 2023-11-24T06:09:17.525 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.517+0000 7fb9e4e1e700 1 Processor -- start 2023-11-24T06:09:17.525 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.517+0000 7fb9e4e1e700 1 -- start start 2023-11-24T06:09:17.525 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9e4e1e700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb9c8003a00 0x7fb9e00b7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.525 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9e4e1e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb9e01f78e0 0x7fb9e01f3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.526 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9e4e1e700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9e00c6f90 0x7fb9e00c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.526 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9e4e1e700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e00b4c70 con 0x7fb9e01f78e0 2023-11-24T06:09:17.526 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9e4e1e700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e00c7950 con 0x7fb9e00c6f90 2023-11-24T06:09:17.526 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9e4e1e700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e00b6ed0 con 0x7fb9c8003a00 2023-11-24T06:09:17.526 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9ddd9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb9e01f78e0 0x7fb9e01f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.526 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9ddd9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb9e01f78e0 0x7fb9e01f3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60112/0 (socket says 172.21.15.91:60112) 2023-11-24T06:09:17.526 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9ddd9b700 1 -- 172.21.15.91:0/2949433153 learned_addr learned my addr 172.21.15.91:0/2949433153 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:17.526 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9ded9d700 1 --2- 172.21.15.91:0/2949433153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9e00c6f90 0x7fb9e00c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.526 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9de59c700 1 --2- 172.21.15.91:0/2949433153 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb9c8003a00 0x7fb9e00b7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.526 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9ddd9b700 1 -- 172.21.15.91:0/2949433153 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb9c8003a00 msgr2=0x7fb9e00b7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.526 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9ddd9b700 1 --2- 172.21.15.91:0/2949433153 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb9c8003a00 0x7fb9e00b7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.526 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9ddd9b700 1 -- 172.21.15.91:0/2949433153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9e00c6f90 msgr2=0x7fb9e00c7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9ddd9b700 1 --2- 172.21.15.91:0/2949433153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9e00c6f90 0x7fb9e00c7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9ddd9b700 1 -- 172.21.15.91:0/2949433153 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e01f37b0 con 0x7fb9e01f78e0 2023-11-24T06:09:17.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9ded9d700 1 --2- 172.21.15.91:0/2949433153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9e00c6f90 0x7fb9e00c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:17.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9de59c700 1 --2- 172.21.15.91:0/2949433153 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb9c8003a00 0x7fb9e00b7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:17.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9ddd9b700 1 --2- 172.21.15.91:0/2949433153 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb9e01f78e0 0x7fb9e01f3270 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00119d0 tx=0x7fb9d0011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:17.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9ceffd700 1 -- 172.21.15.91:0/2949433153 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d0009630 con 0x7fb9e01f78e0 2023-11-24T06:09:17.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9ceffd700 1 -- 172.21.15.91:0/2949433153 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d0004240 con 0x7fb9e01f78e0 2023-11-24T06:09:17.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9e4e1e700 1 -- 172.21.15.91:0/2949433153 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e00b6910 con 0x7fb9e01f78e0 2023-11-24T06:09:17.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9ceffd700 1 -- 172.21.15.91:0/2949433153 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d000f730 con 0x7fb9e01f78e0 2023-11-24T06:09:17.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.518+0000 7fb9e4e1e700 1 -- 172.21.15.91:0/2949433153 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e00b4890 con 0x7fb9e01f78e0 2023-11-24T06:09:17.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.519+0000 7fb9e4e1e700 1 -- 172.21.15.91:0/2949433153 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e00b4ee0 con 0x7fb9e01f78e0 2023-11-24T06:09:17.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.520+0000 7fb9ceffd700 1 -- 172.21.15.91:0/2949433153 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fb9d000f890 con 0x7fb9e01f78e0 2023-11-24T06:09:17.527 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.520+0000 7fb9ceffd700 1 --2- 172.21.15.91:0/2949433153 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb9c006af60 0x7fb9c006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.528 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.521+0000 7fb9de59c700 1 --2- 172.21.15.91:0/2949433153 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb9c006af60 0x7fb9c006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.528 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.521+0000 7fb9ceffd700 1 -- 172.21.15.91:0/2949433153 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9d0089b00 con 0x7fb9e01f78e0 2023-11-24T06:09:17.528 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.521+0000 7fb9de59c700 1 --2- 172.21.15.91:0/2949433153 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb9c006af60 0x7fb9c006d420 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4015040 tx=0x7fb9d4007000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:17.528 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.525+0000 7fb9ceffd700 1 -- 172.21.15.91:0/2949433153 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb9d0054bc0 con 0x7fb9e01f78e0 2023-11-24T06:09:17.716 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.713+0000 7fb9e4e1e700 1 -- 172.21.15.91:0/2949433153 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9e0068670 con 0x7fb9e01f78e0 2023-11-24T06:09:17.717 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.715+0000 7fb9ceffd700 1 -- 172.21.15.91:0/2949433153 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16485 (secure 0 0 0) 0x7fb9d0023090 con 0x7fb9e01f78e0 2023-11-24T06:09:17.718 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:17.718 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":34,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:09:00.042725+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:08:56.059841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:08:57.408988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:09:17.720 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.717+0000 7fb9e4e1e700 1 -- 172.21.15.91:0/2949433153 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb9c006af60 msgr2=0x7fb9c006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.720 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.717+0000 7fb9e4e1e700 1 --2- 172.21.15.91:0/2949433153 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb9c006af60 0x7fb9c006d420 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4015040 tx=0x7fb9d4007000 comp rx=0 tx=0).stop 2023-11-24T06:09:17.720 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.717+0000 7fb9e4e1e700 1 -- 172.21.15.91:0/2949433153 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb9e01f78e0 msgr2=0x7fb9e01f3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.720 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.717+0000 7fb9e4e1e700 1 --2- 172.21.15.91:0/2949433153 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb9e01f78e0 0x7fb9e01f3270 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00119d0 tx=0x7fb9d0011d90 comp rx=0 tx=0).stop 2023-11-24T06:09:17.720 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.717+0000 7fb9e4e1e700 1 -- 172.21.15.91:0/2949433153 shutdown_connections 2023-11-24T06:09:17.720 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.717+0000 7fb9e4e1e700 1 --2- 172.21.15.91:0/2949433153 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fb9c006af60 0x7fb9c006d420 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.720 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.717+0000 7fb9e4e1e700 1 --2- 172.21.15.91:0/2949433153 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fb9e00c6f90 0x7fb9e00c7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.717+0000 7fb9e4e1e700 1 --2- 172.21.15.91:0/2949433153 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fb9c8003a00 0x7fb9e00b7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.717+0000 7fb9e4e1e700 1 --2- 172.21.15.91:0/2949433153 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fb9e01f78e0 0x7fb9e01f3270 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.717+0000 7fb9e4e1e700 1 -- 172.21.15.91:0/2949433153 >> 172.21.15.91:0/2949433153 conn(0x7fb9e00b3720 msgr2=0x7fb9e006ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:17.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.717+0000 7fb9e4e1e700 1 -- 172.21.15.91:0/2949433153 shutdown_connections 2023-11-24T06:09:17.721 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.717+0000 7fb9e4e1e700 1 -- 172.21.15.91:0/2949433153 wait complete. 2023-11-24T06:09:17.732 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-24T06:07:47.435506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-11-24T06:08:56.059841+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-11-24T06:08:57.408988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-24T06:09:17.733 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-11-24T06:09:17.856 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.853+0000 7ffb40203700 1 Processor -- start 2023-11-24T06:09:17.856 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.853+0000 7ffb40203700 1 -- start start 2023-11-24T06:09:17.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.854+0000 7ffb40203700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffb38143ae0 0x7ffb380b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.854+0000 7ffb40203700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ffb380722c0 0x7ffb3806e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.854+0000 7ffb40203700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb38143ec0 con 0x7ffb381440f0 2023-11-24T06:09:17.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.854+0000 7ffb40203700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb380b40a0 con 0x7ffb380722c0 2023-11-24T06:09:17.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.854+0000 7ffb40203700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb380b3ae0 con 0x7ffb38143ae0 2023-11-24T06:09:17.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.854+0000 7ffb3d79e700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7ffb381440f0 0x7ffb38071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55052/0 (socket says 172.21.15.91:55052) 2023-11-24T06:09:17.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.854+0000 7ffb3d79e700 1 -- 172.21.15.91:0/2457773077 learned_addr learned my addr 172.21.15.91:0/2457773077 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:17.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.854+0000 7ffb3df9f700 1 --2- 172.21.15.91:0/2457773077 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffb38143ae0 0x7ffb380b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.854+0000 7ffb3e7a0700 1 --2- 172.21.15.91:0/2457773077 >> v2:172.21.15.94:3300/0 conn(0x7ffb380722c0 0x7ffb3806e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.855+0000 7ffb3cf9d700 1 -- 172.21.15.91:0/2457773077 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729065667 0 0) 0x7ffb38143ec0 con 0x7ffb381440f0 2023-11-24T06:09:17.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.855+0000 7ffb3cf9d700 1 -- 172.21.15.91:0/2457773077 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb200035b0 con 0x7ffb381440f0 2023-11-24T06:09:17.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.855+0000 7ffb3df9f700 1 -- 172.21.15.91:0/2457773077 >> v2:172.21.15.94:3300/0 conn(0x7ffb380722c0 msgr2=0x7ffb3806e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:17.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.855+0000 7ffb3df9f700 1 --2- 172.21.15.91:0/2457773077 >> v2:172.21.15.94:3300/0 conn(0x7ffb380722c0 0x7ffb3806e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.855+0000 7ffb3df9f700 1 -- 172.21.15.91:0/2457773077 >> v1:172.21.15.91:6789/0 conn(0x7ffb381440f0 legacy=0x7ffb38071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.855+0000 7ffb3df9f700 1 -- 172.21.15.91:0/2457773077 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb380b3dc0 con 0x7ffb38143ae0 2023-11-24T06:09:17.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.855+0000 7ffb3e7a0700 1 --2- 172.21.15.91:0/2457773077 >> v2:172.21.15.94:3300/0 conn(0x7ffb380722c0 0x7ffb3806e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:09:17.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.855+0000 7ffb3df9f700 1 --2- 172.21.15.91:0/2457773077 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffb38143ae0 0x7ffb380b51b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb30009d90 tx=0x7ffb3000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94e766f8f4979342 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:17.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.855+0000 7ffb3cf9d700 1 -- 172.21.15.91:0/2457773077 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3000e070 con 0x7ffb38143ae0 2023-11-24T06:09:17.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.855+0000 7ffb3cf9d700 1 -- 172.21.15.91:0/2457773077 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3000bb20 con 0x7ffb38143ae0 2023-11-24T06:09:17.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.855+0000 7ffb3cf9d700 1 -- 172.21.15.91:0/2457773077 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3000be90 con 0x7ffb38143ae0 2023-11-24T06:09:17.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.856+0000 7ffb40203700 1 -- 172.21.15.91:0/2457773077 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffb38143ae0 msgr2=0x7ffb380b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.856+0000 7ffb40203700 1 --2- 172.21.15.91:0/2457773077 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffb38143ae0 0x7ffb380b51b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb30009d90 tx=0x7ffb3000b330 comp rx=0 tx=0).stop 2023-11-24T06:09:17.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.856+0000 7ffb40203700 1 -- 172.21.15.91:0/2457773077 shutdown_connections 2023-11-24T06:09:17.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.856+0000 7ffb40203700 1 --2- 172.21.15.91:0/2457773077 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffb38143ae0 0x7ffb380b51b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.856+0000 7ffb40203700 1 --2- 172.21.15.91:0/2457773077 >> v2:172.21.15.94:3300/0 conn(0x7ffb380722c0 0x7ffb3806e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.856+0000 7ffb40203700 1 -- 172.21.15.91:0/2457773077 >> 172.21.15.91:0/2457773077 conn(0x7ffb380b2740 msgr2=0x7ffb380b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:17.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.856+0000 7ffb40203700 1 -- 172.21.15.91:0/2457773077 shutdown_connections 2023-11-24T06:09:17.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.856+0000 7ffb40203700 1 -- 172.21.15.91:0/2457773077 wait complete. 2023-11-24T06:09:17.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.857+0000 7ffb40203700 1 Processor -- start 2023-11-24T06:09:17.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.857+0000 7ffb40203700 1 -- start start 2023-11-24T06:09:17.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb40203700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb380722c0 0x7ffb3813f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb40203700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ffb38143ae0 0x7ffb38151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb40203700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffb381440f0 0x7ffb381f1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb40203700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb38152070 con 0x7ffb38143ae0 2023-11-24T06:09:17.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb40203700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb38151990 con 0x7ffb380722c0 2023-11-24T06:09:17.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb40203700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb38152b20 con 0x7ffb381440f0 2023-11-24T06:09:17.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb3d79e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ffb38143ae0 0x7ffb38151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb3d79e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ffb38143ae0 0x7ffb38151b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60126/0 (socket says 172.21.15.91:60126) 2023-11-24T06:09:17.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb3d79e700 1 -- 172.21.15.91:0/3139722559 learned_addr learned my addr 172.21.15.91:0/3139722559 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:17.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb3e7a0700 1 --2- 172.21.15.91:0/3139722559 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffb381440f0 0x7ffb381f1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb3df9f700 1 --2- 172.21.15.91:0/3139722559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb380722c0 0x7ffb3813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb3d79e700 1 -- 172.21.15.91:0/3139722559 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffb381440f0 msgr2=0x7ffb381f1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb3d79e700 1 --2- 172.21.15.91:0/3139722559 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffb381440f0 0x7ffb381f1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb3d79e700 1 -- 172.21.15.91:0/3139722559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb380722c0 msgr2=0x7ffb3813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:17.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb3d79e700 1 --2- 172.21.15.91:0/3139722559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb380722c0 0x7ffb3813f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:17.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb3d79e700 1 -- 172.21.15.91:0/3139722559 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb280094a0 con 0x7ffb38143ae0 2023-11-24T06:09:17.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb3d79e700 1 --2- 172.21.15.91:0/3139722559 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ffb38143ae0 0x7ffb38151b30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c00e770 tx=0x7ffb2c00eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:17.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb367fc700 1 -- 172.21.15.91:0/3139722559 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c005d70 con 0x7ffb38143ae0 2023-11-24T06:09:17.866 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb367fc700 1 -- 172.21.15.91:0/3139722559 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2c007070 con 0x7ffb38143ae0 2023-11-24T06:09:17.866 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb40203700 1 -- 172.21.15.91:0/3139722559 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb30009a70 con 0x7ffb38143ae0 2023-11-24T06:09:17.866 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb367fc700 1 -- 172.21.15.91:0/3139722559 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c007420 con 0x7ffb38143ae0 2023-11-24T06:09:17.866 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.858+0000 7ffb40203700 1 -- 172.21.15.91:0/3139722559 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb38152280 con 0x7ffb38143ae0 2023-11-24T06:09:17.866 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.859+0000 7ffb40203700 1 -- 172.21.15.91:0/3139722559 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb38156100 con 0x7ffb38143ae0 2023-11-24T06:09:17.866 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.860+0000 7ffb367fc700 1 -- 172.21.15.91:0/3139722559 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7ffb2c007580 con 0x7ffb38143ae0 2023-11-24T06:09:17.866 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.860+0000 7ffb367fc700 1 --2- 172.21.15.91:0/3139722559 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ffb2406af60 0x7ffb2406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:17.866 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.860+0000 7ffb367fc700 1 -- 172.21.15.91:0/3139722559 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb2c08b9b0 con 0x7ffb38143ae0 2023-11-24T06:09:17.866 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.860+0000 7ffb3df9f700 1 --2- 172.21.15.91:0/3139722559 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ffb2406af60 0x7ffb2406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:17.866 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.860+0000 7ffb3df9f700 1 --2- 172.21.15.91:0/3139722559 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ffb2406af60 0x7ffb2406d420 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ffb30014040 tx=0x7ffb30011000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:17.867 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:17.864+0000 7ffb367fc700 1 -- 172.21.15.91:0/3139722559 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffb2c056960 con 0x7ffb38143ae0 2023-11-24T06:09:18.058 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.055+0000 7ffb40203700 1 -- 172.21.15.91:0/3139722559 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffb38075480 con 0x7ffb38143ae0 2023-11-24T06:09:18.058 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.055+0000 7ffb367fc700 1 -- 172.21.15.91:0/3139722559 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffb2c01f020 con 0x7ffb38143ae0 2023-11-24T06:09:18.058 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 1 2023-11-24T06:09:18.060 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.057+0000 7ffb40203700 1 -- 172.21.15.91:0/3139722559 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ffb2406af60 msgr2=0x7ffb2406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.057+0000 7ffb40203700 1 --2- 172.21.15.91:0/3139722559 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ffb2406af60 0x7ffb2406d420 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ffb30014040 tx=0x7ffb30011000 comp rx=0 tx=0).stop 2023-11-24T06:09:18.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.057+0000 7ffb40203700 1 -- 172.21.15.91:0/3139722559 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ffb38143ae0 msgr2=0x7ffb38151b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.057+0000 7ffb40203700 1 --2- 172.21.15.91:0/3139722559 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ffb38143ae0 0x7ffb38151b30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c00e770 tx=0x7ffb2c00eb30 comp rx=0 tx=0).stop 2023-11-24T06:09:18.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.057+0000 7ffb40203700 1 -- 172.21.15.91:0/3139722559 shutdown_connections 2023-11-24T06:09:18.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.057+0000 7ffb40203700 1 --2- 172.21.15.91:0/3139722559 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ffb2406af60 0x7ffb2406d420 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.057+0000 7ffb40203700 1 --2- 172.21.15.91:0/3139722559 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ffb380722c0 0x7ffb3813f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.057+0000 7ffb40203700 1 --2- 172.21.15.91:0/3139722559 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ffb381440f0 0x7ffb381f1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.057+0000 7ffb40203700 1 --2- 172.21.15.91:0/3139722559 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ffb38143ae0 0x7ffb38151b30 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.057+0000 7ffb40203700 1 -- 172.21.15.91:0/3139722559 >> 172.21.15.91:0/3139722559 conn(0x7ffb380b2740 msgr2=0x7ffb380cfd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:18.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.057+0000 7ffb40203700 1 -- 172.21.15.91:0/3139722559 shutdown_connections 2023-11-24T06:09:18.061 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.057+0000 7ffb40203700 1 -- 172.21.15.91:0/3139722559 wait complete. 2023-11-24T06:09:18.072 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-11-24T06:09:18.195 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.192+0000 7fa8c5b0d700 1 Processor -- start 2023-11-24T06:09:18.195 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.192+0000 7fa8c5b0d700 1 -- start start 2023-11-24T06:09:18.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.193+0000 7fa8c5b0d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8c015bde0 0x7fa8c015c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.193+0000 7fa8c5b0d700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c00d0a90 con 0x7fa8c00cf2b0 2023-11-24T06:09:18.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.193+0000 7fa8c5b0d700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c006ee10 con 0x7fa8c00ce5c0 2023-11-24T06:09:18.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.193+0000 7fa8c5b0d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c006f0f0 con 0x7fa8c015bde0 2023-11-24T06:09:18.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.193+0000 7fa8beffd700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fa8c00cf2b0 0x7fa8c00d01b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55066/0 (socket says 172.21.15.91:55066) 2023-11-24T06:09:18.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.193+0000 7fa8beffd700 1 -- 172.21.15.91:0/2397364844 learned_addr learned my addr 172.21.15.91:0/2397364844 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:18.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.194+0000 7fa8bf7fe700 1 --2- 172.21.15.91:0/2397364844 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8c015bde0 0x7fa8c015c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.194+0000 7fa8be7fc700 1 -- 172.21.15.91:0/2397364844 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476139021 0 0) 0x7fa8c00d0a90 con 0x7fa8c00cf2b0 2023-11-24T06:09:18.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.194+0000 7fa8be7fc700 1 -- 172.21.15.91:0/2397364844 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a80035b0 con 0x7fa8c00cf2b0 2023-11-24T06:09:18.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.194+0000 7fa8bf7fe700 1 -- 172.21.15.91:0/2397364844 >> v1:172.21.15.94:6789/0 conn(0x7fa8c00ce5c0 legacy=0x7fa8c006fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.194+0000 7fa8be7fc700 1 -- 172.21.15.91:0/2397364844 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3614687104 0 0) 0x7fa8a80035b0 con 0x7fa8c00cf2b0 2023-11-24T06:09:18.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.194+0000 7fa8bf7fe700 1 -- 172.21.15.91:0/2397364844 >> v1:172.21.15.91:6789/0 conn(0x7fa8c00cf2b0 legacy=0x7fa8c00d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.194+0000 7fa8bf7fe700 1 -- 172.21.15.91:0/2397364844 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c006f3d0 con 0x7fa8c015bde0 2023-11-24T06:09:18.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.194+0000 7fa8bf7fe700 1 --2- 172.21.15.91:0/2397364844 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8c015bde0 0x7fa8c015c1c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009870 tx=0x7fa8b4009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36ecef0a9ed0342c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:18.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.194+0000 7fa8be7fc700 1 -- 172.21.15.91:0/2397364844 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0001670 con 0x7fa8c015bde0 2023-11-24T06:09:18.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.194+0000 7fa8be7fc700 1 -- 172.21.15.91:0/2397364844 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b4004030 con 0x7fa8c015bde0 2023-11-24T06:09:18.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.194+0000 7fa8be7fc700 1 -- 172.21.15.91:0/2397364844 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b40043c0 con 0x7fa8c015bde0 2023-11-24T06:09:18.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.195+0000 7fa8c5b0d700 1 -- 172.21.15.91:0/2397364844 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8c015bde0 msgr2=0x7fa8c015c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.195+0000 7fa8c5b0d700 1 --2- 172.21.15.91:0/2397364844 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8c015bde0 0x7fa8c015c1c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009870 tx=0x7fa8b4009b80 comp rx=0 tx=0).stop 2023-11-24T06:09:18.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.195+0000 7fa8c5b0d700 1 -- 172.21.15.91:0/2397364844 shutdown_connections 2023-11-24T06:09:18.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.195+0000 7fa8c5b0d700 1 --2- 172.21.15.91:0/2397364844 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8c015bde0 0x7fa8c015c1c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.195+0000 7fa8c5b0d700 1 -- 172.21.15.91:0/2397364844 >> 172.21.15.91:0/2397364844 conn(0x7fa8c00b3720 msgr2=0x7fa8c00b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:18.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.195+0000 7fa8c5b0d700 1 -- 172.21.15.91:0/2397364844 shutdown_connections 2023-11-24T06:09:18.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.195+0000 7fa8c5b0d700 1 -- 172.21.15.91:0/2397364844 wait complete. 2023-11-24T06:09:18.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.196+0000 7fa8c5b0d700 1 Processor -- start 2023-11-24T06:09:18.202 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.196+0000 7fa8c5b0d700 1 -- start start 2023-11-24T06:09:18.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8c5b0d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8c00ce5c0 0x7fa8c009f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8c5b0d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8c00cf2b0 0x7fa8c014e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8c5b0d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa8c015bde0 0x7fa8c015c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8c5b0d700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c015d740 con 0x7fa8c015bde0 2023-11-24T06:09:18.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8c5b0d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c015cb30 con 0x7fa8c00cf2b0 2023-11-24T06:09:18.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8c5b0d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c015ce10 con 0x7fa8c00ce5c0 2023-11-24T06:09:18.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8bffff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa8c015bde0 0x7fa8c015c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8bffff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa8c015bde0 0x7fa8c015c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60142/0 (socket says 172.21.15.91:60142) 2023-11-24T06:09:18.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8bffff700 1 -- 172.21.15.91:0/172718492 learned_addr learned my addr 172.21.15.91:0/172718492 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:18.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8bf7fe700 1 --2- 172.21.15.91:0/172718492 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8c00ce5c0 0x7fa8c009f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8beffd700 1 --2- 172.21.15.91:0/172718492 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8c00cf2b0 0x7fa8c014e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8bffff700 1 -- 172.21.15.91:0/172718492 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8c00ce5c0 msgr2=0x7fa8c009f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8bffff700 1 --2- 172.21.15.91:0/172718492 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8c00ce5c0 0x7fa8c009f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8bffff700 1 -- 172.21.15.91:0/172718492 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8c00cf2b0 msgr2=0x7fa8c014e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8bffff700 1 --2- 172.21.15.91:0/172718492 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8c00cf2b0 0x7fa8c014e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8bffff700 1 -- 172.21.15.91:0/172718492 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b40094a0 con 0x7fa8c015bde0 2023-11-24T06:09:18.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.197+0000 7fa8bffff700 1 --2- 172.21.15.91:0/172718492 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa8c015bde0 0x7fa8c015c1c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa8b800c890 tx=0x7fa8b800cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:18.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.198+0000 7fa8a7fff700 1 -- 172.21.15.91:0/172718492 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8003e30 con 0x7fa8c015bde0 2023-11-24T06:09:18.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.198+0000 7fa8a7fff700 1 -- 172.21.15.91:0/172718492 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b8005070 con 0x7fa8c015bde0 2023-11-24T06:09:18.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.198+0000 7fa8c5b0d700 1 -- 172.21.15.91:0/172718492 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c015d0f0 con 0x7fa8c015bde0 2023-11-24T06:09:18.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.198+0000 7fa8a7fff700 1 -- 172.21.15.91:0/172718492 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8005420 con 0x7fa8c015bde0 2023-11-24T06:09:18.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.198+0000 7fa8c5b0d700 1 -- 172.21.15.91:0/172718492 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c014eea0 con 0x7fa8c015bde0 2023-11-24T06:09:18.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.198+0000 7fa8c5b0d700 1 -- 172.21.15.91:0/172718492 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c00cee30 con 0x7fa8c015bde0 2023-11-24T06:09:18.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.199+0000 7fa8a7fff700 1 -- 172.21.15.91:0/172718492 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fa8b8005580 con 0x7fa8c015bde0 2023-11-24T06:09:18.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.199+0000 7fa8a7fff700 1 --2- 172.21.15.91:0/172718492 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa8ac06af60 0x7fa8ac06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.199+0000 7fa8a7fff700 1 -- 172.21.15.91:0/172718492 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8b8089300 con 0x7fa8c015bde0 2023-11-24T06:09:18.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.199+0000 7fa8bf7fe700 1 --2- 172.21.15.91:0/172718492 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa8ac06af60 0x7fa8ac06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.200+0000 7fa8bf7fe700 1 --2- 172.21.15.91:0/172718492 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa8ac06af60 0x7fa8ac06d420 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa8b40097c0 tx=0x7fa8b4008000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:18.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.204+0000 7fa8a7fff700 1 -- 172.21.15.91:0/172718492 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa8b80543c0 con 0x7fa8c015bde0 2023-11-24T06:09:18.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.393+0000 7fa8c5b0d700 1 -- 172.21.15.91:0/172718492 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa8c00cffd0 con 0x7fa8c015bde0 2023-11-24T06:09:18.396 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.393+0000 7fa8a7fff700 1 -- 172.21.15.91:0/172718492 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa8b80579e0 con 0x7fa8c015bde0 2023-11-24T06:09:18.396 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 32 2023-11-24T06:09:18.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.395+0000 7fa8c5b0d700 1 -- 172.21.15.91:0/172718492 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa8ac06af60 msgr2=0x7fa8ac06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.398 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.395+0000 7fa8c5b0d700 1 --2- 172.21.15.91:0/172718492 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa8ac06af60 0x7fa8ac06d420 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa8b40097c0 tx=0x7fa8b4008000 comp rx=0 tx=0).stop 2023-11-24T06:09:18.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.395+0000 7fa8c5b0d700 1 -- 172.21.15.91:0/172718492 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa8c015bde0 msgr2=0x7fa8c015c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.395+0000 7fa8c5b0d700 1 --2- 172.21.15.91:0/172718492 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa8c015bde0 0x7fa8c015c1c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa8b800c890 tx=0x7fa8b800cba0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.395+0000 7fa8c5b0d700 1 -- 172.21.15.91:0/172718492 shutdown_connections 2023-11-24T06:09:18.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.395+0000 7fa8c5b0d700 1 --2- 172.21.15.91:0/172718492 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa8ac06af60 0x7fa8ac06d420 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.395+0000 7fa8c5b0d700 1 --2- 172.21.15.91:0/172718492 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa8c015bde0 0x7fa8c015c1c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.395+0000 7fa8c5b0d700 1 --2- 172.21.15.91:0/172718492 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa8c00ce5c0 0x7fa8c009f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.395+0000 7fa8c5b0d700 1 --2- 172.21.15.91:0/172718492 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa8c00cf2b0 0x7fa8c014e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.395+0000 7fa8c5b0d700 1 -- 172.21.15.91:0/172718492 >> 172.21.15.91:0/172718492 conn(0x7fa8c00b3720 msgr2=0x7fa8c00b2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:18.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.395+0000 7fa8c5b0d700 1 -- 172.21.15.91:0/172718492 shutdown_connections 2023-11-24T06:09:18.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.395+0000 7fa8c5b0d700 1 -- 172.21.15.91:0/172718492 wait complete. 2023-11-24T06:09:18.410 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-11-24T06:09:18.535 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.532+0000 7f946d72d700 1 Processor -- start 2023-11-24T06:09:18.535 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.532+0000 7f946d72d700 1 -- start start 2023-11-24T06:09:18.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.533+0000 7f946d72d700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f94680d0870 0x7f94680d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.533+0000 7f946d72d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f94680d0c50 0x7f94680d49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.533+0000 7f946d72d700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94680b5370 con 0x7f94680cf860 2023-11-24T06:09:18.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.533+0000 7f946d72d700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94680d2f80 con 0x7f94680d0c50 2023-11-24T06:09:18.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.533+0000 7f946d72d700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94680ceae0 con 0x7f94680d0870 2023-11-24T06:09:18.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.533+0000 7f94667fc700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f94680d0c50 0x7f94680d49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.533+0000 7f94677fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f94680cf860 0x7f94680b6d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55080/0 (socket says 172.21.15.91:55080) 2023-11-24T06:09:18.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.533+0000 7f94677fe700 1 -- 172.21.15.91:0/3301093755 learned_addr learned my addr 172.21.15.91:0/3301093755 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:18.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.533+0000 7f9466ffd700 1 --2- 172.21.15.91:0/3301093755 >> v2:172.21.15.94:3300/0 conn(0x7f94680d0870 0x7f94680d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.533+0000 7f94667fc700 1 -- 172.21.15.91:0/3301093755 >> v2:172.21.15.94:3300/0 conn(0x7f94680d0870 msgr2=0x7f94680d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:18.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.533+0000 7f94667fc700 1 --2- 172.21.15.91:0/3301093755 >> v2:172.21.15.94:3300/0 conn(0x7f94680d0870 0x7f94680d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.538 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.533+0000 7f94667fc700 1 -- 172.21.15.91:0/3301093755 >> v1:172.21.15.91:6789/0 conn(0x7f94680cf860 legacy=0x7f94680b6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.533+0000 7f94667fc700 1 -- 172.21.15.91:0/3301093755 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94680d2ca0 con 0x7f94680d0c50 2023-11-24T06:09:18.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.533+0000 7f9465ffb700 1 -- 172.21.15.91:0/3301093755 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876023967 0 0) 0x7f94680b5370 con 0x7f94680cf860 2023-11-24T06:09:18.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.534+0000 7f94667fc700 1 --2- 172.21.15.91:0/3301093755 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94680d0c50 0x7f94680d49f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f946808c560 tx=0x7f9458009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52f656000962461 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:18.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.534+0000 7f9465ffb700 1 -- 172.21.15.91:0/3301093755 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9458011830 con 0x7f94680d0c50 2023-11-24T06:09:18.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.534+0000 7f9465ffb700 1 -- 172.21.15.91:0/3301093755 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9458011990 con 0x7f94680d0c50 2023-11-24T06:09:18.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.534+0000 7f9465ffb700 1 -- 172.21.15.91:0/3301093755 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9458011d20 con 0x7f94680d0c50 2023-11-24T06:09:18.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.534+0000 7f946d72d700 1 -- 172.21.15.91:0/3301093755 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94680d0c50 msgr2=0x7f94680d49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.534+0000 7f946d72d700 1 --2- 172.21.15.91:0/3301093755 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94680d0c50 0x7f94680d49f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f946808c560 tx=0x7f9458009b20 comp rx=0 tx=0).stop 2023-11-24T06:09:18.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.534+0000 7f946d72d700 1 -- 172.21.15.91:0/3301093755 shutdown_connections 2023-11-24T06:09:18.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.534+0000 7f946d72d700 1 --2- 172.21.15.91:0/3301093755 >> v2:172.21.15.94:3300/0 conn(0x7f94680d0870 0x7f94680d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.534+0000 7f946d72d700 1 --2- 172.21.15.91:0/3301093755 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94680d0c50 0x7f94680d49f0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.539 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.534+0000 7f946d72d700 1 -- 172.21.15.91:0/3301093755 >> 172.21.15.91:0/3301093755 conn(0x7f94680d3ed0 msgr2=0x7f94680d42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:18.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.534+0000 7f946d72d700 1 -- 172.21.15.91:0/3301093755 shutdown_connections 2023-11-24T06:09:18.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.534+0000 7f946d72d700 1 -- 172.21.15.91:0/3301093755 wait complete. 2023-11-24T06:09:18.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.536+0000 7f946d72d700 1 Processor -- start 2023-11-24T06:09:18.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.536+0000 7f946d72d700 1 -- start start 2023-11-24T06:09:18.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.536+0000 7f946d72d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94680cf860 0x7f94680a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.536+0000 7f946d72d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f94680d0870 0x7f94680a0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.536+0000 7f946d72d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94680d0c50 0x7f94681445d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.536+0000 7f946d72d700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9468157170 con 0x7f94680cf860 2023-11-24T06:09:18.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.536+0000 7f946d72d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94680a0780 con 0x7f94680d0c50 2023-11-24T06:09:18.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.536+0000 7f946d72d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94680a0a60 con 0x7f94680d0870 2023-11-24T06:09:18.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f9466ffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94680cf860 0x7f94680a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f9466ffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94680cf860 0x7f94680a1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60150/0 (socket says 172.21.15.91:60150) 2023-11-24T06:09:18.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f9466ffd700 1 -- 172.21.15.91:0/3141123433 learned_addr learned my addr 172.21.15.91:0/3141123433 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:18.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f94677fe700 1 --2- 172.21.15.91:0/3141123433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94680d0c50 0x7f94681445d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f94667fc700 1 --2- 172.21.15.91:0/3141123433 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f94680d0870 0x7f94680a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f9466ffd700 1 -- 172.21.15.91:0/3141123433 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f94680d0870 msgr2=0x7f94680a0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f9466ffd700 1 --2- 172.21.15.91:0/3141123433 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f94680d0870 0x7f94680a0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f9466ffd700 1 -- 172.21.15.91:0/3141123433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94680d0c50 msgr2=0x7f94681445d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f9466ffd700 1 --2- 172.21.15.91:0/3141123433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94680d0c50 0x7f94681445d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f9466ffd700 1 -- 172.21.15.91:0/3141123433 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94580094a0 con 0x7f94680cf860 2023-11-24T06:09:18.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f94667fc700 1 --2- 172.21.15.91:0/3141123433 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f94680d0870 0x7f94680a0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:18.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f9466ffd700 1 --2- 172.21.15.91:0/3141123433 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94680cf860 0x7f94680a1820 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f945c00e870 tx=0x7f945c00eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:18.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f944f7fe700 1 -- 172.21.15.91:0/3141123433 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c009590 con 0x7f94680cf860 2023-11-24T06:09:18.545 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f944f7fe700 1 -- 172.21.15.91:0/3141123433 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945c013680 con 0x7f94680cf860 2023-11-24T06:09:18.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f946d72d700 1 -- 172.21.15.91:0/3141123433 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94680a0d40 con 0x7f94680cf860 2023-11-24T06:09:18.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f944f7fe700 1 -- 172.21.15.91:0/3141123433 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c0139f0 con 0x7f94680cf860 2023-11-24T06:09:18.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.537+0000 7f946d72d700 1 -- 172.21.15.91:0/3141123433 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94680a1300 con 0x7f94680cf860 2023-11-24T06:09:18.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.538+0000 7f946d72d700 1 -- 172.21.15.91:0/3141123433 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94680b57c0 con 0x7f94680cf860 2023-11-24T06:09:18.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.539+0000 7f944f7fe700 1 -- 172.21.15.91:0/3141123433 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f945c013b50 con 0x7f94680cf860 2023-11-24T06:09:18.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.539+0000 7f944f7fe700 1 --2- 172.21.15.91:0/3141123433 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f945406af60 0x7f945406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.539+0000 7f944f7fe700 1 -- 172.21.15.91:0/3141123433 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f945c089470 con 0x7f94680cf860 2023-11-24T06:09:18.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.539+0000 7f94667fc700 1 --2- 172.21.15.91:0/3141123433 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f945406af60 0x7f945406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.546 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.539+0000 7f94667fc700 1 --2- 172.21.15.91:0/3141123433 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f945406af60 0x7f945406d420 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f946808c560 tx=0x7f9458008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:18.547 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.543+0000 7f944f7fe700 1 -- 172.21.15.91:0/3141123433 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f945c0545c0 con 0x7f94680cf860 2023-11-24T06:09:18.734 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.732+0000 7f946d72d700 1 -- 172.21.15.91:0/3141123433 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9468067a20 con 0x7f94680cf860 2023-11-24T06:09:18.782 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.732+0000 7f944f7fe700 1 -- 172.21.15.91:0/3141123433 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7f945c057be0 con 0x7f94680cf860 2023-11-24T06:09:18.783 INFO:teuthology.orchestra.run.smithi091.stdout:pg_num: 64 2023-11-24T06:09:18.785 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.734+0000 7f946d72d700 1 -- 172.21.15.91:0/3141123433 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f945406af60 msgr2=0x7f945406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.785 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.734+0000 7f946d72d700 1 --2- 172.21.15.91:0/3141123433 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f945406af60 0x7f945406d420 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f946808c560 tx=0x7f9458008030 comp rx=0 tx=0).stop 2023-11-24T06:09:18.785 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.734+0000 7f946d72d700 1 -- 172.21.15.91:0/3141123433 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94680cf860 msgr2=0x7f94680a1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.785 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.734+0000 7f946d72d700 1 --2- 172.21.15.91:0/3141123433 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94680cf860 0x7f94680a1820 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f945c00e870 tx=0x7f945c00eb80 comp rx=0 tx=0).stop 2023-11-24T06:09:18.785 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.734+0000 7f946d72d700 1 -- 172.21.15.91:0/3141123433 shutdown_connections 2023-11-24T06:09:18.786 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.734+0000 7f946d72d700 1 --2- 172.21.15.91:0/3141123433 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f945406af60 0x7f945406d420 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.786 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.734+0000 7f946d72d700 1 --2- 172.21.15.91:0/3141123433 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f94680d0c50 0x7f94681445d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.786 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.734+0000 7f946d72d700 1 --2- 172.21.15.91:0/3141123433 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f94680cf860 0x7f94680a1820 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.786 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.734+0000 7f946d72d700 1 --2- 172.21.15.91:0/3141123433 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f94680d0870 0x7f94680a0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.786 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.734+0000 7f946d72d700 1 -- 172.21.15.91:0/3141123433 >> 172.21.15.91:0/3141123433 conn(0x7f94680d3ed0 msgr2=0x7f9468159600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:18.786 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.734+0000 7f946d72d700 1 -- 172.21.15.91:0/3141123433 shutdown_connections 2023-11-24T06:09:18.786 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.734+0000 7f946d72d700 1 -- 172.21.15.91:0/3141123433 wait complete. 2023-11-24T06:09:18.787 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:09:18.907 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.904+0000 7f7277fd5700 1 Processor -- start 2023-11-24T06:09:18.907 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.904+0000 7f7277fd5700 1 -- start start 2023-11-24T06:09:18.909 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.905+0000 7f7277fd5700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f72701448b0 0x7f727006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.905+0000 7f7277fd5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72700b44c0 0x7f727006f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.905+0000 7f7277fd5700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7270144c90 con 0x7f727006f7c0 2023-11-24T06:09:18.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.905+0000 7f7277fd5700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f727006fc50 con 0x7f72701448b0 2023-11-24T06:09:18.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.905+0000 7f7277fd5700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72700b4c80 con 0x7f72700b44c0 2023-11-24T06:09:18.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.906+0000 7f7275d71700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f72701448b0 0x7f727006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.906+0000 7f7275d71700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f72701448b0 0x7f727006ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:44550/0 (socket says 172.21.15.91:44550) 2023-11-24T06:09:18.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.906+0000 7f7275d71700 1 -- 172.21.15.91:0/471293336 learned_addr learned my addr 172.21.15.91:0/471293336 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:18.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.906+0000 7f7275570700 1 --2- 172.21.15.91:0/471293336 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72700b44c0 0x7f727006f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.910 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.906+0000 7f7275d71700 1 -- 172.21.15.91:0/471293336 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72700b44c0 msgr2=0x7f727006f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:18.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.906+0000 7f7275d71700 1 --2- 172.21.15.91:0/471293336 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72700b44c0 0x7f727006f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.906+0000 7f7275d71700 1 -- 172.21.15.91:0/471293336 >> v1:172.21.15.94:6789/0 conn(0x7f727006f7c0 legacy=0x7f727006db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.906+0000 7f7275d71700 1 -- 172.21.15.91:0/471293336 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72700b4f60 con 0x7f72701448b0 2023-11-24T06:09:18.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.906+0000 7f7275d71700 1 --2- 172.21.15.91:0/471293336 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72701448b0 0x7f727006ec70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f72600098f0 tx=0x7f7260009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f97ca452e6425c7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:18.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.906+0000 7f7274d6f700 1 -- 172.21.15.91:0/471293336 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7260011830 con 0x7f72701448b0 2023-11-24T06:09:18.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.906+0000 7f7274d6f700 1 -- 172.21.15.91:0/471293336 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7260011990 con 0x7f72701448b0 2023-11-24T06:09:18.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.906+0000 7f7274d6f700 1 -- 172.21.15.91:0/471293336 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7260011d20 con 0x7f72701448b0 2023-11-24T06:09:18.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.907+0000 7f7277fd5700 1 -- 172.21.15.91:0/471293336 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72701448b0 msgr2=0x7f727006ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.907+0000 7f7277fd5700 1 --2- 172.21.15.91:0/471293336 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72701448b0 0x7f727006ec70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f72600098f0 tx=0x7f7260009c00 comp rx=0 tx=0).stop 2023-11-24T06:09:18.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.907+0000 7f7277fd5700 1 -- 172.21.15.91:0/471293336 shutdown_connections 2023-11-24T06:09:18.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.907+0000 7f7277fd5700 1 --2- 172.21.15.91:0/471293336 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f72701448b0 0x7f727006ec70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.911 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.907+0000 7f7277fd5700 1 --2- 172.21.15.91:0/471293336 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72700b44c0 0x7f727006f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.907+0000 7f7277fd5700 1 -- 172.21.15.91:0/471293336 >> 172.21.15.91:0/471293336 conn(0x7f72700d0bc0 msgr2=0x7f72700d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:18.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.907+0000 7f7277fd5700 1 -- 172.21.15.91:0/471293336 shutdown_connections 2023-11-24T06:09:18.912 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.907+0000 7f7277fd5700 1 -- 172.21.15.91:0/471293336 wait complete. 2023-11-24T06:09:18.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.908+0000 7f7277fd5700 1 Processor -- start 2023-11-24T06:09:18.914 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.908+0000 7f7277fd5700 1 -- start start 2023-11-24T06:09:18.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7277fd5700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f727006f7c0 0x7f727015a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7277fd5700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72700b44c0 0x7f727013f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7277fd5700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72701448b0 0x7f727013fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7277fd5700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727015d5b0 con 0x7f72701448b0 2023-11-24T06:09:18.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7277fd5700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72701402b0 con 0x7f727006f7c0 2023-11-24T06:09:18.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7277fd5700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7270140590 con 0x7f72700b44c0 2023-11-24T06:09:18.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7276572700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72701448b0 0x7f727013fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7276572700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72701448b0 0x7f727013fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60168/0 (socket says 172.21.15.91:60168) 2023-11-24T06:09:18.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7276572700 1 -- 172.21.15.91:0/373195864 learned_addr learned my addr 172.21.15.91:0/373195864 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:18.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7275d71700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f727006f7c0 0x7f727015a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.915 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7275570700 1 --2- 172.21.15.91:0/373195864 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72700b44c0 0x7f727013f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7276572700 1 -- 172.21.15.91:0/373195864 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72700b44c0 msgr2=0x7f727013f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7276572700 1 --2- 172.21.15.91:0/373195864 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72700b44c0 0x7f727013f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7276572700 1 -- 172.21.15.91:0/373195864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f727006f7c0 msgr2=0x7f727015a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:18.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7276572700 1 --2- 172.21.15.91:0/373195864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f727006f7c0 0x7f727015a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:18.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7276572700 1 -- 172.21.15.91:0/373195864 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7260009550 con 0x7f72701448b0 2023-11-24T06:09:18.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7275d71700 1 --2- 172.21.15.91:0/373195864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f727006f7c0 0x7f727015a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:09:18.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.909+0000 7f7276572700 1 --2- 172.21.15.91:0/373195864 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72701448b0 0x7f727013fcc0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f726800cc50 tx=0x7f726800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:18.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.910+0000 7f72667fc700 1 -- 172.21.15.91:0/373195864 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7268003c00 con 0x7f72701448b0 2023-11-24T06:09:18.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.910+0000 7f72667fc700 1 -- 172.21.15.91:0/373195864 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7268003d60 con 0x7f72701448b0 2023-11-24T06:09:18.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.910+0000 7f7277fd5700 1 -- 172.21.15.91:0/373195864 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7270140870 con 0x7f72701448b0 2023-11-24T06:09:18.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.910+0000 7f72667fc700 1 -- 172.21.15.91:0/373195864 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726800aab0 con 0x7f72701448b0 2023-11-24T06:09:18.916 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.910+0000 7f7277fd5700 1 -- 172.21.15.91:0/373195864 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7270140e30 con 0x7f72701448b0 2023-11-24T06:09:18.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.911+0000 7f7277fd5700 1 -- 172.21.15.91:0/373195864 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7270068670 con 0x7f72701448b0 2023-11-24T06:09:18.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.911+0000 7f72667fc700 1 -- 172.21.15.91:0/373195864 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f726800ac10 con 0x7f72701448b0 2023-11-24T06:09:18.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.911+0000 7f72667fc700 1 --2- 172.21.15.91:0/373195864 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f725406af90 0x7f725406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:18.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.911+0000 7f72667fc700 1 -- 172.21.15.91:0/373195864 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7268089550 con 0x7f72701448b0 2023-11-24T06:09:18.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.912+0000 7f7275d71700 1 --2- 172.21.15.91:0/373195864 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f725406af90 0x7f725406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:18.917 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.912+0000 7f7275d71700 1 --2- 172.21.15.91:0/373195864 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f725406af90 0x7f725406d450 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7260000c00 tx=0x7f7260008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:18.919 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:18.916+0000 7f72667fc700 1 -- 172.21.15.91:0/373195864 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f72680546a0 con 0x7f72701448b0 2023-11-24T06:09:19.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.148+0000 7f7277fd5700 1 -- 172.21.15.91:0/373195864 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72700cf540 con 0x7f72701448b0 2023-11-24T06:09:19.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.149+0000 7f72667fc700 1 -- 172.21.15.91:0/373195864 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4907 (secure 0 0 0) 0x7f7268057cc0 con 0x7f72701448b0 2023-11-24T06:09:19.152 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:19.152 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4706,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1685174039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1685174039},{"type":"v1","addr":"172.21.15.94:6834","nonce":1685174039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4709,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1508344631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1508344631},{"type":"v1","addr":"172.21.15.94:6835","nonce":1508344631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5064,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6836/28114058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6834","nonce":28114058},{"type":"v1","addr":"172.21.15.91:6836","nonce":28114058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:08:58.437951+0000","modified":"2023-11-24T06:09:01.476634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.91:6837/1301930708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":1301930708},{"type":"v1","addr":"172.21.15.91:6837","nonce":1301930708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T06:09:19.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.151+0000 7f7277fd5700 1 -- 172.21.15.91:0/373195864 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f725406af90 msgr2=0x7f725406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:19.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.151+0000 7f7277fd5700 1 --2- 172.21.15.91:0/373195864 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f725406af90 0x7f725406d450 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7260000c00 tx=0x7f7260008030 comp rx=0 tx=0).stop 2023-11-24T06:09:19.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.151+0000 7f7277fd5700 1 -- 172.21.15.91:0/373195864 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72701448b0 msgr2=0x7f727013fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:19.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.151+0000 7f7277fd5700 1 --2- 172.21.15.91:0/373195864 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72701448b0 0x7f727013fcc0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f726800cc50 tx=0x7f726800cf60 comp rx=0 tx=0).stop 2023-11-24T06:09:19.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.151+0000 7f7277fd5700 1 -- 172.21.15.91:0/373195864 shutdown_connections 2023-11-24T06:09:19.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.151+0000 7f7277fd5700 1 --2- 172.21.15.91:0/373195864 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f725406af90 0x7f725406d450 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.151+0000 7f7277fd5700 1 --2- 172.21.15.91:0/373195864 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f727006f7c0 0x7f727015a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.151+0000 7f7277fd5700 1 --2- 172.21.15.91:0/373195864 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f72701448b0 0x7f727013fcc0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.151+0000 7f7277fd5700 1 --2- 172.21.15.91:0/373195864 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f72700b44c0 0x7f727013f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.151+0000 7f7277fd5700 1 -- 172.21.15.91:0/373195864 >> 172.21.15.91:0/373195864 conn(0x7f72700d0bc0 msgr2=0x7f72700afec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:19.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.151+0000 7f7277fd5700 1 -- 172.21.15.91:0/373195864 shutdown_connections 2023-11-24T06:09:19.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.151+0000 7f7277fd5700 1 -- 172.21.15.91:0/373195864 wait complete. 2023-11-24T06:09:19.155 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 17 2023-11-24T06:09:19.166 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:09:19.285 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.283+0000 7f7d45c22700 1 Processor -- start 2023-11-24T06:09:19.286 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.283+0000 7f7d45c22700 1 -- start start 2023-11-24T06:09:19.288 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d45c22700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7d4006b6e0 0x7f7d4006db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:19.288 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d45c22700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f7d40145100 0x7f7d4006e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:19.288 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d45c22700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d40160720 con 0x7f7d40144c10 2023-11-24T06:09:19.288 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d45c22700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4006f500 con 0x7f7d40145100 2023-11-24T06:09:19.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d45c22700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d400cd610 con 0x7f7d4006b6e0 2023-11-24T06:09:19.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d3ffff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f7d40145100 0x7f7d4006e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:19.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d3ffff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f7d40145100 0x7f7d4006e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60176/0 (socket says 172.21.15.91:60176) 2023-11-24T06:09:19.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d3ffff700 1 -- 172.21.15.91:0/2768602611 learned_addr learned my addr 172.21.15.91:0/2768602611 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:19.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d3effd700 1 --2- 172.21.15.91:0/2768602611 >> v2:172.21.15.94:3300/0 conn(0x7f7d4006b6e0 0x7f7d4006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:19.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d3ffff700 1 -- 172.21.15.91:0/2768602611 >> v2:172.21.15.94:3300/0 conn(0x7f7d4006b6e0 msgr2=0x7f7d4006db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:19.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d3ffff700 1 --2- 172.21.15.91:0/2768602611 >> v2:172.21.15.94:3300/0 conn(0x7f7d4006b6e0 0x7f7d4006db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d3ffff700 1 -- 172.21.15.91:0/2768602611 >> v1:172.21.15.94:6789/0 conn(0x7f7d40144c10 legacy=0x7f7d4006afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:19.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d3ffff700 1 -- 172.21.15.91:0/2768602611 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4006f710 con 0x7f7d40145100 2023-11-24T06:09:19.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d3ffff700 1 --2- 172.21.15.91:0/2768602611 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7d40145100 0x7f7d4006e0b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7d40071b60 tx=0x7f7d3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463bb6e1dbd99052 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:19.289 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d3e7fc700 1 -- 172.21.15.91:0/2768602611 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3800e070 con 0x7f7d40145100 2023-11-24T06:09:19.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d3e7fc700 1 -- 172.21.15.91:0/2768602611 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d38003a20 con 0x7f7d40145100 2023-11-24T06:09:19.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.284+0000 7f7d3e7fc700 1 -- 172.21.15.91:0/2768602611 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d38003dd0 con 0x7f7d40145100 2023-11-24T06:09:19.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.285+0000 7f7d45c22700 1 -- 172.21.15.91:0/2768602611 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7d40145100 msgr2=0x7f7d4006e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:19.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.285+0000 7f7d45c22700 1 --2- 172.21.15.91:0/2768602611 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7d40145100 0x7f7d4006e0b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7d40071b60 tx=0x7f7d3800b330 comp rx=0 tx=0).stop 2023-11-24T06:09:19.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.285+0000 7f7d45c22700 1 -- 172.21.15.91:0/2768602611 shutdown_connections 2023-11-24T06:09:19.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.285+0000 7f7d45c22700 1 --2- 172.21.15.91:0/2768602611 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7d40145100 0x7f7d4006e0b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.285+0000 7f7d45c22700 1 --2- 172.21.15.91:0/2768602611 >> v2:172.21.15.94:3300/0 conn(0x7f7d4006b6e0 0x7f7d4006db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.285+0000 7f7d45c22700 1 -- 172.21.15.91:0/2768602611 >> 172.21.15.91:0/2768602611 conn(0x7f7d400b2740 msgr2=0x7f7d400b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:19.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.285+0000 7f7d45c22700 1 -- 172.21.15.91:0/2768602611 shutdown_connections 2023-11-24T06:09:19.290 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.285+0000 7f7d45c22700 1 -- 172.21.15.91:0/2768602611 wait complete. 2023-11-24T06:09:19.293 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.286+0000 7f7d45c22700 1 Processor -- start 2023-11-24T06:09:19.293 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.287+0000 7f7d45c22700 1 -- start start 2023-11-24T06:09:19.293 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d45c22700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7d4006b6e0 0x7f7d4013a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:19.293 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d45c22700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7d40144c10 0x7f7d4013b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:19.294 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d45c22700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7d40145100 0x7f7d401f1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:19.294 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d45c22700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4013c030 con 0x7f7d40144c10 2023-11-24T06:09:19.294 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d45c22700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4013faa0 con 0x7f7d40145100 2023-11-24T06:09:19.294 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d45c22700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4013fd80 con 0x7f7d4006b6e0 2023-11-24T06:09:19.294 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d3effd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7d40144c10 0x7f7d4013b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:19.294 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d3effd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7d40144c10 0x7f7d4013b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60180/0 (socket says 172.21.15.91:60180) 2023-11-24T06:09:19.294 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d3effd700 1 -- 172.21.15.91:0/2917597106 learned_addr learned my addr 172.21.15.91:0/2917597106 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:19.294 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d3ffff700 1 --2- 172.21.15.91:0/2917597106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7d40145100 0x7f7d401f1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:19.294 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d3f7fe700 1 --2- 172.21.15.91:0/2917597106 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7d4006b6e0 0x7f7d4013a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:19.294 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d3effd700 1 -- 172.21.15.91:0/2917597106 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7d4006b6e0 msgr2=0x7f7d4013a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:19.294 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d3effd700 1 --2- 172.21.15.91:0/2917597106 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7d4006b6e0 0x7f7d4013a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.295 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d3effd700 1 -- 172.21.15.91:0/2917597106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7d40145100 msgr2=0x7f7d401f1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:19.295 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d3effd700 1 --2- 172.21.15.91:0/2917597106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7d40145100 0x7f7d401f1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.295 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d3effd700 1 -- 172.21.15.91:0/2917597106 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d38009b30 con 0x7f7d40144c10 2023-11-24T06:09:19.295 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d3ffff700 1 --2- 172.21.15.91:0/2917597106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7d40145100 0x7f7d401f1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:19.295 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d3f7fe700 1 --2- 172.21.15.91:0/2917597106 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7d4006b6e0 0x7f7d4013a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:19.295 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d3effd700 1 --2- 172.21.15.91:0/2917597106 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7d40144c10 0x7f7d4013b800 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400c7f0 tx=0x7f7d3400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:19.295 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d27fff700 1 -- 172.21.15.91:0/2917597106 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34011800 con 0x7f7d40144c10 2023-11-24T06:09:19.295 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d27fff700 1 -- 172.21.15.91:0/2917597106 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d34011960 con 0x7f7d40144c10 2023-11-24T06:09:19.295 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d45c22700 1 -- 172.21.15.91:0/2917597106 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d40140060 con 0x7f7d40144c10 2023-11-24T06:09:19.295 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d27fff700 1 -- 172.21.15.91:0/2917597106 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34011cf0 con 0x7f7d40144c10 2023-11-24T06:09:19.295 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.288+0000 7f7d45c22700 1 -- 172.21.15.91:0/2917597106 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4013c310 con 0x7f7d40144c10 2023-11-24T06:09:19.295 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.289+0000 7f7d45c22700 1 -- 172.21.15.91:0/2917597106 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d401403d0 con 0x7f7d40144c10 2023-11-24T06:09:19.296 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.290+0000 7f7d27fff700 1 -- 172.21.15.91:0/2917597106 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f7d34018360 con 0x7f7d40144c10 2023-11-24T06:09:19.296 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.290+0000 7f7d27fff700 1 --2- 172.21.15.91:0/2917597106 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7d2c06af60 0x7f7d2c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:19.296 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.290+0000 7f7d27fff700 1 -- 172.21.15.91:0/2917597106 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d34089700 con 0x7f7d40144c10 2023-11-24T06:09:19.296 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.290+0000 7f7d3f7fe700 1 --2- 172.21.15.91:0/2917597106 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7d2c06af60 0x7f7d2c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:19.296 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.290+0000 7f7d3f7fe700 1 --2- 172.21.15.91:0/2917597106 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7d2c06af60 0x7f7d2c06d420 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7d30001750 tx=0x7f7d30009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:19.297 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.294+0000 7f7d27fff700 1 -- 172.21.15.91:0/2917597106 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d340546c0 con 0x7f7d40144c10 2023-11-24T06:09:19.490 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.487+0000 7f7d45c22700 1 -- 172.21.15.91:0/2917597106 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d40068670 con 0x7f7d40144c10 2023-11-24T06:09:19.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.492+0000 7f7d27fff700 1 -- 172.21.15.91:0/2917597106 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16485 (secure 0 0 0) 0x7f7d34057ce0 con 0x7f7d40144c10 2023-11-24T06:09:19.495 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:19.495 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":34,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:09:00.042725+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:08:56.059841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:08:57.408988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:09:19.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.494+0000 7f7d45c22700 1 -- 172.21.15.91:0/2917597106 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7d2c06af60 msgr2=0x7f7d2c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:19.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.494+0000 7f7d45c22700 1 --2- 172.21.15.91:0/2917597106 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7d2c06af60 0x7f7d2c06d420 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7d30001750 tx=0x7f7d30009040 comp rx=0 tx=0).stop 2023-11-24T06:09:19.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.494+0000 7f7d45c22700 1 -- 172.21.15.91:0/2917597106 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7d40144c10 msgr2=0x7f7d4013b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:19.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.494+0000 7f7d45c22700 1 --2- 172.21.15.91:0/2917597106 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7d40144c10 0x7f7d4013b800 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400c7f0 tx=0x7f7d3400cbb0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.494+0000 7f7d45c22700 1 -- 172.21.15.91:0/2917597106 shutdown_connections 2023-11-24T06:09:19.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.494+0000 7f7d45c22700 1 --2- 172.21.15.91:0/2917597106 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f7d2c06af60 0x7f7d2c06d420 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.494+0000 7f7d45c22700 1 --2- 172.21.15.91:0/2917597106 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7d40145100 0x7f7d401f1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.494+0000 7f7d45c22700 1 --2- 172.21.15.91:0/2917597106 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f7d4006b6e0 0x7f7d4013a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.494+0000 7f7d45c22700 1 --2- 172.21.15.91:0/2917597106 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f7d40144c10 0x7f7d4013b800 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.494+0000 7f7d45c22700 1 -- 172.21.15.91:0/2917597106 >> 172.21.15.91:0/2917597106 conn(0x7f7d400b2740 msgr2=0x7f7d400723e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:19.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.494+0000 7f7d45c22700 1 -- 172.21.15.91:0/2917597106 shutdown_connections 2023-11-24T06:09:19.498 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.494+0000 7f7d45c22700 1 -- 172.21.15.91:0/2917597106 wait complete. 2023-11-24T06:09:19.509 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T06:09:19.509 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:09:19.629 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.626+0000 7f0b8ddce700 1 Processor -- start 2023-11-24T06:09:19.629 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.626+0000 7f0b8ddce700 1 -- start start 2023-11-24T06:09:19.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.627+0000 7f0b8ddce700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f0b880b4150 0x7f0b8806de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:19.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.627+0000 7f0b8ddce700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0b880b5ab0 0x7f0b8806e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:19.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.627+0000 7f0b8ddce700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b88071f50 con 0x7f0b8806b620 2023-11-24T06:09:19.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.627+0000 7f0b8ddce700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b88072230 con 0x7f0b880b4150 2023-11-24T06:09:19.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.627+0000 7f0b8ddce700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b88072510 con 0x7f0b880b5ab0 2023-11-24T06:09:19.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.627+0000 7f0b86ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f0b880b4150 0x7f0b8806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:19.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.627+0000 7f0b877fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f0b8806b620 0x7f0b880b3a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55120/0 (socket says 172.21.15.91:55120) 2023-11-24T06:09:19.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.627+0000 7f0b86ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f0b880b4150 0x7f0b8806de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60186/0 (socket says 172.21.15.91:60186) 2023-11-24T06:09:19.632 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.627+0000 7f0b877fe700 1 -- 172.21.15.91:0/362331274 learned_addr learned my addr 172.21.15.91:0/362331274 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:19.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.628+0000 7f0b87fff700 1 --2- 172.21.15.91:0/362331274 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0b880b5ab0 0x7f0b8806e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:19.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.628+0000 7f0b86ffd700 1 -- 172.21.15.91:0/362331274 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0b880b5ab0 msgr2=0x7f0b8806e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:19.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.628+0000 7f0b86ffd700 1 --2- 172.21.15.91:0/362331274 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0b880b5ab0 0x7f0b8806e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.628+0000 7f0b86ffd700 1 -- 172.21.15.91:0/362331274 >> v1:172.21.15.91:6789/0 conn(0x7f0b8806b620 legacy=0x7f0b880b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:19.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.628+0000 7f0b867fc700 1 -- 172.21.15.91:0/362331274 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429597029 0 0) 0x7f0b88071f50 con 0x7f0b8806b620 2023-11-24T06:09:19.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.628+0000 7f0b86ffd700 1 -- 172.21.15.91:0/362331274 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b880727f0 con 0x7f0b880b4150 2023-11-24T06:09:19.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.628+0000 7f0b86ffd700 1 --2- 172.21.15.91:0/362331274 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0b880b4150 0x7f0b8806de50 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009840 tx=0x7f0b70009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe3cab4a89554d0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:19.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.628+0000 7f0b867fc700 1 -- 172.21.15.91:0/362331274 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70011830 con 0x7f0b880b4150 2023-11-24T06:09:19.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.628+0000 7f0b867fc700 1 -- 172.21.15.91:0/362331274 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b70011990 con 0x7f0b880b4150 2023-11-24T06:09:19.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.628+0000 7f0b867fc700 1 -- 172.21.15.91:0/362331274 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70011d20 con 0x7f0b880b4150 2023-11-24T06:09:19.633 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.629+0000 7f0b8ddce700 1 -- 172.21.15.91:0/362331274 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0b880b4150 msgr2=0x7f0b8806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:19.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.629+0000 7f0b8ddce700 1 --2- 172.21.15.91:0/362331274 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0b880b4150 0x7f0b8806de50 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009840 tx=0x7f0b70009b50 comp rx=0 tx=0).stop 2023-11-24T06:09:19.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.629+0000 7f0b8ddce700 1 -- 172.21.15.91:0/362331274 shutdown_connections 2023-11-24T06:09:19.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.629+0000 7f0b8ddce700 1 --2- 172.21.15.91:0/362331274 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0b880b5ab0 0x7f0b8806e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.629+0000 7f0b8ddce700 1 --2- 172.21.15.91:0/362331274 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0b880b4150 0x7f0b8806de50 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.629+0000 7f0b8ddce700 1 -- 172.21.15.91:0/362331274 >> 172.21.15.91:0/362331274 conn(0x7f0b880b2740 msgr2=0x7f0b880b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:19.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.629+0000 7f0b8ddce700 1 -- 172.21.15.91:0/362331274 shutdown_connections 2023-11-24T06:09:19.634 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.629+0000 7f0b8ddce700 1 -- 172.21.15.91:0/362331274 wait complete. 2023-11-24T06:09:19.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.630+0000 7f0b8ddce700 1 Processor -- start 2023-11-24T06:09:19.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.630+0000 7f0b8ddce700 1 -- start start 2023-11-24T06:09:19.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.631+0000 7f0b8ddce700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b8806b620 0x7f0b881f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:19.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.631+0000 7f0b8ddce700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0b880b4150 0x7f0b8806eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:19.637 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.631+0000 7f0b8ddce700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0b880b5ab0 0x7f0b8806f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:19.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.631+0000 7f0b8ddce700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b881412c0 con 0x7f0b880b5ab0 2023-11-24T06:09:19.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.631+0000 7f0b8ddce700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b881f3700 con 0x7f0b8806b620 2023-11-24T06:09:19.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.631+0000 7f0b8ddce700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b88140220 con 0x7f0b880b4150 2023-11-24T06:09:19.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.631+0000 7f0b87fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0b880b5ab0 0x7f0b8806f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:19.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.631+0000 7f0b87fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0b880b5ab0 0x7f0b8806f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60196/0 (socket says 172.21.15.91:60196) 2023-11-24T06:09:19.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.631+0000 7f0b87fff700 1 -- 172.21.15.91:0/497483847 learned_addr learned my addr 172.21.15.91:0/497483847 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:19.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.632+0000 7f0b877fe700 1 --2- 172.21.15.91:0/497483847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b8806b620 0x7f0b881f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:19.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.632+0000 7f0b86ffd700 1 --2- 172.21.15.91:0/497483847 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0b880b4150 0x7f0b8806eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:19.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.632+0000 7f0b87fff700 1 -- 172.21.15.91:0/497483847 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0b880b4150 msgr2=0x7f0b8806eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:19.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.632+0000 7f0b87fff700 1 --2- 172.21.15.91:0/497483847 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0b880b4150 0x7f0b8806eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.638 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.632+0000 7f0b87fff700 1 -- 172.21.15.91:0/497483847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b8806b620 msgr2=0x7f0b881f31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:19.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.632+0000 7f0b87fff700 1 --2- 172.21.15.91:0/497483847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b8806b620 0x7f0b881f31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.632+0000 7f0b87fff700 1 -- 172.21.15.91:0/497483847 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b700094a0 con 0x7f0b880b5ab0 2023-11-24T06:09:19.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.632+0000 7f0b86ffd700 1 --2- 172.21.15.91:0/497483847 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0b880b4150 0x7f0b8806eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:19.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.632+0000 7f0b87fff700 1 --2- 172.21.15.91:0/497483847 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0b880b5ab0 0x7f0b8806f410 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0b80007e40 tx=0x7f0b8000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:19.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.632+0000 7f0b7ffff700 1 -- 172.21.15.91:0/497483847 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8000ede0 con 0x7f0b880b5ab0 2023-11-24T06:09:19.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.632+0000 7f0b7ffff700 1 -- 172.21.15.91:0/497483847 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b80014420 con 0x7f0b880b5ab0 2023-11-24T06:09:19.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.632+0000 7f0b8ddce700 1 -- 172.21.15.91:0/497483847 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b88140500 con 0x7f0b880b5ab0 2023-11-24T06:09:19.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.632+0000 7f0b7ffff700 1 -- 172.21.15.91:0/497483847 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b800147d0 con 0x7f0b880b5ab0 2023-11-24T06:09:19.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.632+0000 7f0b8ddce700 1 -- 172.21.15.91:0/497483847 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b88140ac0 con 0x7f0b880b5ab0 2023-11-24T06:09:19.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.633+0000 7f0b8ddce700 1 -- 172.21.15.91:0/497483847 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b88068670 con 0x7f0b880b5ab0 2023-11-24T06:09:19.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.633+0000 7f0b7ffff700 1 -- 172.21.15.91:0/497483847 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f0b80014930 con 0x7f0b880b5ab0 2023-11-24T06:09:19.639 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.634+0000 7f0b7ffff700 1 --2- 172.21.15.91:0/497483847 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0b6806af60 0x7f0b6806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:19.640 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.634+0000 7f0b7ffff700 1 -- 172.21.15.91:0/497483847 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b80089d40 con 0x7f0b880b5ab0 2023-11-24T06:09:19.640 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.634+0000 7f0b877fe700 1 --2- 172.21.15.91:0/497483847 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0b6806af60 0x7f0b6806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:19.640 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.634+0000 7f0b877fe700 1 --2- 172.21.15.91:0/497483847 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0b6806af60 0x7f0b6806d420 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0b78000f60 tx=0x7f0b78007e70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:19.641 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.638+0000 7f0b7ffff700 1 -- 172.21.15.91:0/497483847 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0b80054e00 con 0x7f0b880b5ab0 2023-11-24T06:09:19.873 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.871+0000 7f0b8ddce700 1 -- 172.21.15.91:0/497483847 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b8806bd40 con 0x7f0b880b5ab0 2023-11-24T06:09:19.874 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.871+0000 7f0b7ffff700 1 -- 172.21.15.91:0/497483847 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4907 (secure 0 0 0) 0x7f0b80058420 con 0x7f0b880b5ab0 2023-11-24T06:09:19.874 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:19.874 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4706,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1685174039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1685174039},{"type":"v1","addr":"172.21.15.94:6834","nonce":1685174039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4709,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1508344631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1508344631},{"type":"v1","addr":"172.21.15.94:6835","nonce":1508344631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5064,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6836/28114058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6834","nonce":28114058},{"type":"v1","addr":"172.21.15.91:6836","nonce":28114058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:08:58.437951+0000","modified":"2023-11-24T06:09:01.476634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.91:6837/1301930708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":1301930708},{"type":"v1","addr":"172.21.15.91:6837","nonce":1301930708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T06:09:19.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.874+0000 7f0b8ddce700 1 -- 172.21.15.91:0/497483847 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0b6806af60 msgr2=0x7f0b6806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:19.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.874+0000 7f0b8ddce700 1 --2- 172.21.15.91:0/497483847 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0b6806af60 0x7f0b6806d420 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0b78000f60 tx=0x7f0b78007e70 comp rx=0 tx=0).stop 2023-11-24T06:09:19.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.874+0000 7f0b8ddce700 1 -- 172.21.15.91:0/497483847 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0b880b5ab0 msgr2=0x7f0b8806f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:19.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.874+0000 7f0b8ddce700 1 --2- 172.21.15.91:0/497483847 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0b880b5ab0 0x7f0b8806f410 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0b80007e40 tx=0x7f0b8000e3c0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.874+0000 7f0b8ddce700 1 -- 172.21.15.91:0/497483847 shutdown_connections 2023-11-24T06:09:19.877 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.874+0000 7f0b8ddce700 1 --2- 172.21.15.91:0/497483847 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0b6806af60 0x7f0b6806d420 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.874+0000 7f0b8ddce700 1 --2- 172.21.15.91:0/497483847 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0b8806b620 0x7f0b881f31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.874+0000 7f0b8ddce700 1 --2- 172.21.15.91:0/497483847 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0b880b5ab0 0x7f0b8806f410 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.874+0000 7f0b8ddce700 1 --2- 172.21.15.91:0/497483847 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0b880b4150 0x7f0b8806eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:19.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.874+0000 7f0b8ddce700 1 -- 172.21.15.91:0/497483847 >> 172.21.15.91:0/497483847 conn(0x7f0b880b2740 msgr2=0x7f0b880affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:19.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.874+0000 7f0b8ddce700 1 -- 172.21.15.91:0/497483847 shutdown_connections 2023-11-24T06:09:19.878 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:19.874+0000 7f0b8ddce700 1 -- 172.21.15.91:0/497483847 wait complete. 2023-11-24T06:09:19.878 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 17 2023-11-24T06:09:19.889 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:09:20.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.007+0000 7f3bfeb81700 1 Processor -- start 2023-11-24T06:09:20.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.007+0000 7f3bfeb81700 1 -- start start 2023-11-24T06:09:20.013 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.008+0000 7f3bfeb81700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f3bf81617a0 0x7f3bf80d23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.013 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.008+0000 7f3bfeb81700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f3bf80d2930 0x7f3bf80b4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.013 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.008+0000 7f3bfeb81700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf80d2d40 con 0x7f3bf80ce9b0 2023-11-24T06:09:20.013 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.008+0000 7f3bfeb81700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf80d4930 con 0x7f3bf80d2930 2023-11-24T06:09:20.013 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.008+0000 7f3bfeb81700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf80d4c10 con 0x7f3bf81617a0 2023-11-24T06:09:20.013 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.008+0000 7f3bf7fff700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f3bf80ce9b0 0x7f3bf8161d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55134/0 (socket says 172.21.15.91:55134) 2023-11-24T06:09:20.014 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.008+0000 7f3bf7fff700 1 -- 172.21.15.91:0/1999661562 learned_addr learned my addr 172.21.15.91:0/1999661562 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:20.014 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.008+0000 7f3bfd11e700 1 --2- 172.21.15.91:0/1999661562 >> v2:172.21.15.91:3300/0 conn(0x7f3bf80d2930 0x7f3bf80b4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.014 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.008+0000 7f3bfc91d700 1 --2- 172.21.15.91:0/1999661562 >> v2:172.21.15.94:3300/0 conn(0x7f3bf81617a0 0x7f3bf80d23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.014 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.008+0000 7f3bfd11e700 1 -- 172.21.15.91:0/1999661562 >> v2:172.21.15.94:3300/0 conn(0x7f3bf81617a0 msgr2=0x7f3bf80d23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:20.014 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.008+0000 7f3bfd11e700 1 --2- 172.21.15.91:0/1999661562 >> v2:172.21.15.94:3300/0 conn(0x7f3bf81617a0 0x7f3bf80d23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.014 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.008+0000 7f3bfd11e700 1 -- 172.21.15.91:0/1999661562 >> v1:172.21.15.91:6789/0 conn(0x7f3bf80ce9b0 legacy=0x7f3bf8161d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:20.014 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.008+0000 7f3bfd11e700 1 -- 172.21.15.91:0/1999661562 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf80d19e0 con 0x7f3bf80d2930 2023-11-24T06:09:20.014 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.008+0000 7f3bf77fe700 1 -- 172.21.15.91:0/1999661562 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286191679 0 0) 0x7f3bf80d2d40 con 0x7f3bf80ce9b0 2023-11-24T06:09:20.014 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.009+0000 7f3bfd11e700 1 --2- 172.21.15.91:0/1999661562 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3bf80d2930 0x7f3bf80b4b60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3bf806b870 tx=0x7f3bf000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3362104888ae8676 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:20.014 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.009+0000 7f3bf77fe700 1 -- 172.21.15.91:0/1999661562 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf000e070 con 0x7f3bf80d2930 2023-11-24T06:09:20.014 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.009+0000 7f3bf77fe700 1 -- 172.21.15.91:0/1999661562 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf000bbd0 con 0x7f3bf80d2930 2023-11-24T06:09:20.014 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.009+0000 7f3bf77fe700 1 -- 172.21.15.91:0/1999661562 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0012430 con 0x7f3bf80d2930 2023-11-24T06:09:20.015 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.009+0000 7f3bfeb81700 1 -- 172.21.15.91:0/1999661562 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3bf80d2930 msgr2=0x7f3bf80b4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:20.015 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.009+0000 7f3bfeb81700 1 --2- 172.21.15.91:0/1999661562 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3bf80d2930 0x7f3bf80b4b60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3bf806b870 tx=0x7f3bf000b330 comp rx=0 tx=0).stop 2023-11-24T06:09:20.015 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.009+0000 7f3bfeb81700 1 -- 172.21.15.91:0/1999661562 shutdown_connections 2023-11-24T06:09:20.015 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.009+0000 7f3bfeb81700 1 --2- 172.21.15.91:0/1999661562 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3bf80d2930 0x7f3bf80b4b60 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.015 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.009+0000 7f3bfeb81700 1 --2- 172.21.15.91:0/1999661562 >> v2:172.21.15.94:3300/0 conn(0x7f3bf81617a0 0x7f3bf80d23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.015 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.009+0000 7f3bfeb81700 1 -- 172.21.15.91:0/1999661562 >> 172.21.15.91:0/1999661562 conn(0x7f3bf80d3e10 msgr2=0x7f3bf80d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:20.015 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.009+0000 7f3bfeb81700 1 -- 172.21.15.91:0/1999661562 shutdown_connections 2023-11-24T06:09:20.015 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.009+0000 7f3bfeb81700 1 -- 172.21.15.91:0/1999661562 wait complete. 2023-11-24T06:09:20.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.011+0000 7f3bfeb81700 1 Processor -- start 2023-11-24T06:09:20.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.011+0000 7f3bfeb81700 1 -- start start 2023-11-24T06:09:20.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.012+0000 7f3bfeb81700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3bf80ce9b0 0x7f3bf80b8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.012+0000 7f3bfeb81700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3bf80d2930 0x7f3bf81561e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.012+0000 7f3bfeb81700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3bf81617a0 0x7f3bf8156720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.012+0000 7f3bfeb81700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf8153d80 con 0x7f3bf80d2930 2023-11-24T06:09:20.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.012+0000 7f3bfeb81700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf80b93d0 con 0x7f3bf80ce9b0 2023-11-24T06:09:20.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.012+0000 7f3bfeb81700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf80b7df0 con 0x7f3bf81617a0 2023-11-24T06:09:20.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.012+0000 7f3bf7fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3bf80d2930 0x7f3bf81561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.012+0000 7f3bf7fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3bf80d2930 0x7f3bf81561e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60206/0 (socket says 172.21.15.91:60206) 2023-11-24T06:09:20.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.012+0000 7f3bf7fff700 1 -- 172.21.15.91:0/957941221 learned_addr learned my addr 172.21.15.91:0/957941221 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:20.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.012+0000 7f3bfd11e700 1 --2- 172.21.15.91:0/957941221 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3bf81617a0 0x7f3bf8156720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.012+0000 7f3bfc91d700 1 --2- 172.21.15.91:0/957941221 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3bf80ce9b0 0x7f3bf80b8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.012+0000 7f3bf7fff700 1 -- 172.21.15.91:0/957941221 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3bf81617a0 msgr2=0x7f3bf8156720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:20.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.012+0000 7f3bf7fff700 1 --2- 172.21.15.91:0/957941221 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3bf81617a0 0x7f3bf8156720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.012+0000 7f3bf7fff700 1 -- 172.21.15.91:0/957941221 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3bf80ce9b0 msgr2=0x7f3bf80b8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:20.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.013+0000 7f3bf7fff700 1 --2- 172.21.15.91:0/957941221 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3bf80ce9b0 0x7f3bf80b8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.013+0000 7f3bf7fff700 1 -- 172.21.15.91:0/957941221 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf0009a70 con 0x7f3bf80d2930 2023-11-24T06:09:20.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.013+0000 7f3bf7fff700 1 --2- 172.21.15.91:0/957941221 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3bf80d2930 0x7f3bf81561e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3be400c520 tx=0x7f3be400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:20.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.013+0000 7f3bf57fa700 1 -- 172.21.15.91:0/957941221 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be4004c80 con 0x7f3bf80d2930 2023-11-24T06:09:20.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.013+0000 7f3bf57fa700 1 -- 172.21.15.91:0/957941221 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be4004de0 con 0x7f3bf80d2930 2023-11-24T06:09:20.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.013+0000 7f3bfeb81700 1 -- 172.21.15.91:0/957941221 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf80b80d0 con 0x7f3bf80d2930 2023-11-24T06:09:20.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.013+0000 7f3bf57fa700 1 -- 172.21.15.91:0/957941221 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be40072d0 con 0x7f3bf80d2930 2023-11-24T06:09:20.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.013+0000 7f3bfeb81700 1 -- 172.21.15.91:0/957941221 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf80b8690 con 0x7f3bf80d2930 2023-11-24T06:09:20.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.014+0000 7f3bfeb81700 1 -- 172.21.15.91:0/957941221 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf80678e0 con 0x7f3bf80d2930 2023-11-24T06:09:20.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.014+0000 7f3bf57fa700 1 -- 172.21.15.91:0/957941221 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f3be4007430 con 0x7f3bf80d2930 2023-11-24T06:09:20.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.015+0000 7f3bf57fa700 1 --2- 172.21.15.91:0/957941221 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3bdc06af60 0x7f3bdc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.015+0000 7f3bf57fa700 1 -- 172.21.15.91:0/957941221 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3be4089320 con 0x7f3bf80d2930 2023-11-24T06:09:20.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.015+0000 7f3bfc91d700 1 --2- 172.21.15.91:0/957941221 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3bdc06af60 0x7f3bdc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.015+0000 7f3bfc91d700 1 --2- 172.21.15.91:0/957941221 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3bdc06af60 0x7f3bdc06d420 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3bec009930 tx=0x7f3bec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:20.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.019+0000 7f3bf57fa700 1 -- 172.21.15.91:0/957941221 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3be40543e0 con 0x7f3bf80d2930 2023-11-24T06:09:20.256 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.254+0000 7f3bfeb81700 1 -- 172.21.15.91:0/957941221 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bf8068670 con 0x7f3bf80d2930 2023-11-24T06:09:20.257 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.254+0000 7f3bf57fa700 1 -- 172.21.15.91:0/957941221 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+4907 (secure 0 0 0) 0x7f3be4057a00 con 0x7f3bf80d2930 2023-11-24T06:09:20.257 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:20.257 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4706,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1685174039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1685174039},{"type":"v1","addr":"172.21.15.94:6834","nonce":1685174039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4709,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1508344631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1508344631},{"type":"v1","addr":"172.21.15.94:6835","nonce":1508344631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5064,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6836/28114058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6834","nonce":28114058},{"type":"v1","addr":"172.21.15.91:6836","nonce":28114058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-11-24T06:08:58.437951+0000","modified":"2023-11-24T06:09:01.476634+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5067},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5067":{"gid":5067,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.91:6837/1301930708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":1301930708},{"type":"v1","addr":"172.21.15.91:6837","nonce":1301930708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-11-24T06:09:20.260 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.257+0000 7f3bfeb81700 1 -- 172.21.15.91:0/957941221 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3bdc06af60 msgr2=0x7f3bdc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:20.260 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.257+0000 7f3bfeb81700 1 --2- 172.21.15.91:0/957941221 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3bdc06af60 0x7f3bdc06d420 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3bec009930 tx=0x7f3bec008040 comp rx=0 tx=0).stop 2023-11-24T06:09:20.260 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.257+0000 7f3bfeb81700 1 -- 172.21.15.91:0/957941221 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3bf80d2930 msgr2=0x7f3bf81561e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:20.260 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.257+0000 7f3bfeb81700 1 --2- 172.21.15.91:0/957941221 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3bf80d2930 0x7f3bf81561e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3be400c520 tx=0x7f3be400c8e0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.260 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.257+0000 7f3bfeb81700 1 -- 172.21.15.91:0/957941221 shutdown_connections 2023-11-24T06:09:20.260 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.257+0000 7f3bfeb81700 1 --2- 172.21.15.91:0/957941221 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f3bdc06af60 0x7f3bdc06d420 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.260 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.257+0000 7f3bfeb81700 1 --2- 172.21.15.91:0/957941221 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f3bf80ce9b0 0x7f3bf80b8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.260 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.257+0000 7f3bfeb81700 1 --2- 172.21.15.91:0/957941221 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f3bf81617a0 0x7f3bf8156720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.260 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.257+0000 7f3bfeb81700 1 --2- 172.21.15.91:0/957941221 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f3bf80d2930 0x7f3bf81561e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.257+0000 7f3bfeb81700 1 -- 172.21.15.91:0/957941221 >> 172.21.15.91:0/957941221 conn(0x7f3bf80d3e10 msgr2=0x7f3bf80d0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:20.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.257+0000 7f3bfeb81700 1 -- 172.21.15.91:0/957941221 shutdown_connections 2023-11-24T06:09:20.261 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.257+0000 7f3bfeb81700 1 -- 172.21.15.91:0/957941221 wait complete. 2023-11-24T06:09:20.261 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 17 2023-11-24T06:09:20.272 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:09:20.391 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.388+0000 7f5b35363700 1 Processor -- start 2023-11-24T06:09:20.391 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.388+0000 7f5b35363700 1 -- start start 2023-11-24T06:09:20.394 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.389+0000 7f5b35363700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5b3006b620 0x7f5b300b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.394 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.389+0000 7f5b35363700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f5b300b5ab0 0x7f5b300b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.394 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.389+0000 7f5b35363700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f5b301478c0 0x7f5b30161ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.394 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.389+0000 7f5b35363700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b30071f50 con 0x7f5b301478c0 2023-11-24T06:09:20.394 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.389+0000 7f5b35363700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b30072230 con 0x7f5b300b5ab0 2023-11-24T06:09:20.394 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.389+0000 7f5b35363700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b30072510 con 0x7f5b3006b620 2023-11-24T06:09:20.394 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.389+0000 7f5b2f7fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f5b301478c0 0x7f5b30161ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.394 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.389+0000 7f5b2f7fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f5b301478c0 0x7f5b30161ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60216/0 (socket says 172.21.15.91:60216) 2023-11-24T06:09:20.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.389+0000 7f5b2f7fe700 1 -- 172.21.15.91:0/1914889559 learned_addr learned my addr 172.21.15.91:0/1914889559 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:20.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.389+0000 7f5b2e7fc700 1 --2- 172.21.15.91:0/1914889559 >> v2:172.21.15.94:3300/0 conn(0x7f5b300b5ab0 0x7f5b300b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.389+0000 7f5b2effd700 1 --2- 172.21.15.91:0/1914889559 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5b3006b620 0x7f5b300b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.390+0000 7f5b2f7fe700 1 -- 172.21.15.91:0/1914889559 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5b3006b620 msgr2=0x7f5b300b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:20.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.390+0000 7f5b2f7fe700 1 --2- 172.21.15.91:0/1914889559 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5b3006b620 0x7f5b300b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.390+0000 7f5b2f7fe700 1 -- 172.21.15.91:0/1914889559 >> v2:172.21.15.94:3300/0 conn(0x7f5b300b5ab0 msgr2=0x7f5b300b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:20.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.390+0000 7f5b2f7fe700 1 --2- 172.21.15.91:0/1914889559 >> v2:172.21.15.94:3300/0 conn(0x7f5b300b5ab0 0x7f5b300b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.390+0000 7f5b2f7fe700 1 -- 172.21.15.91:0/1914889559 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b300727f0 con 0x7f5b301478c0 2023-11-24T06:09:20.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.390+0000 7f5b2f7fe700 1 --2- 172.21.15.91:0/1914889559 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5b301478c0 0x7f5b30161ba0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5b30069550 tx=0x7f5b2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807098a43eaa5ab2 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:20.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.390+0000 7f5b2dffb700 1 -- 172.21.15.91:0/1914889559 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2800e070 con 0x7f5b301478c0 2023-11-24T06:09:20.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.390+0000 7f5b2dffb700 1 -- 172.21.15.91:0/1914889559 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2800b910 con 0x7f5b301478c0 2023-11-24T06:09:20.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.390+0000 7f5b2dffb700 1 -- 172.21.15.91:0/1914889559 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2800bcc0 con 0x7f5b301478c0 2023-11-24T06:09:20.395 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.390+0000 7f5b35363700 1 -- 172.21.15.91:0/1914889559 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5b301478c0 msgr2=0x7f5b30161ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:20.396 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.391+0000 7f5b35363700 1 --2- 172.21.15.91:0/1914889559 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5b301478c0 0x7f5b30161ba0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5b30069550 tx=0x7f5b2800b330 comp rx=0 tx=0).stop 2023-11-24T06:09:20.396 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.391+0000 7f5b35363700 1 -- 172.21.15.91:0/1914889559 shutdown_connections 2023-11-24T06:09:20.396 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.391+0000 7f5b35363700 1 --2- 172.21.15.91:0/1914889559 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5b301478c0 0x7f5b30161ba0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.396 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.391+0000 7f5b35363700 1 --2- 172.21.15.91:0/1914889559 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5b3006b620 0x7f5b300b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.396 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.391+0000 7f5b35363700 1 --2- 172.21.15.91:0/1914889559 >> v2:172.21.15.94:3300/0 conn(0x7f5b300b5ab0 0x7f5b300b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.396 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.391+0000 7f5b35363700 1 -- 172.21.15.91:0/1914889559 >> 172.21.15.91:0/1914889559 conn(0x7f5b300b2740 msgr2=0x7f5b300b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:20.396 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.391+0000 7f5b35363700 1 -- 172.21.15.91:0/1914889559 shutdown_connections 2023-11-24T06:09:20.396 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.391+0000 7f5b35363700 1 -- 172.21.15.91:0/1914889559 wait complete. 2023-11-24T06:09:20.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.392+0000 7f5b35363700 1 Processor -- start 2023-11-24T06:09:20.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.392+0000 7f5b35363700 1 -- start start 2023-11-24T06:09:20.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b35363700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5b3006b620 0x7f5b300cc810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b35363700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5b300b5ab0 0x7f5b300c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b35363700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b301478c0 0x7f5b300c9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b35363700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300c6350 con 0x7f5b3006b620 2023-11-24T06:09:20.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b35363700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300c9510 con 0x7f5b301478c0 2023-11-24T06:09:20.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b35363700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b300c59b0 con 0x7f5b300b5ab0 2023-11-24T06:09:20.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b2effd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5b3006b620 0x7f5b300cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b2effd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5b3006b620 0x7f5b300cc810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60218/0 (socket says 172.21.15.91:60218) 2023-11-24T06:09:20.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b2effd700 1 -- 172.21.15.91:0/1326340830 learned_addr learned my addr 172.21.15.91:0/1326340830 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:20.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b2e7fc700 1 --2- 172.21.15.91:0/1326340830 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5b300b5ab0 0x7f5b300c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b2f7fe700 1 --2- 172.21.15.91:0/1326340830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b301478c0 0x7f5b300c9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b2effd700 1 -- 172.21.15.91:0/1326340830 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5b300b5ab0 msgr2=0x7f5b300c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:20.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b2effd700 1 --2- 172.21.15.91:0/1326340830 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5b300b5ab0 0x7f5b300c9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b2effd700 1 -- 172.21.15.91:0/1326340830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b301478c0 msgr2=0x7f5b300c9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:20.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b2effd700 1 --2- 172.21.15.91:0/1326340830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b301478c0 0x7f5b300c9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b2effd700 1 -- 172.21.15.91:0/1326340830 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b28009b30 con 0x7f5b3006b620 2023-11-24T06:09:20.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b2effd700 1 --2- 172.21.15.91:0/1326340830 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5b3006b620 0x7f5b300cc810 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400e930 tx=0x7f5b2400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:20.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b177fe700 1 -- 172.21.15.91:0/1326340830 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b24013680 con 0x7f5b3006b620 2023-11-24T06:09:20.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b177fe700 1 -- 172.21.15.91:0/1326340830 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b240137e0 con 0x7f5b3006b620 2023-11-24T06:09:20.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.393+0000 7f5b35363700 1 -- 172.21.15.91:0/1326340830 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b300c5c90 con 0x7f5b3006b620 2023-11-24T06:09:20.401 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.394+0000 7f5b177fe700 1 -- 172.21.15.91:0/1326340830 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b24013b70 con 0x7f5b3006b620 2023-11-24T06:09:20.401 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.394+0000 7f5b35363700 1 -- 172.21.15.91:0/1326340830 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b30145bb0 con 0x7f5b3006b620 2023-11-24T06:09:20.401 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.395+0000 7f5b35363700 1 -- 172.21.15.91:0/1326340830 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b300b54d0 con 0x7f5b3006b620 2023-11-24T06:09:20.401 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.395+0000 7f5b177fe700 1 -- 172.21.15.91:0/1326340830 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f5b24013cd0 con 0x7f5b3006b620 2023-11-24T06:09:20.401 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.395+0000 7f5b177fe700 1 --2- 172.21.15.91:0/1326340830 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f5b1c06af60 0x7f5b1c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.401 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.395+0000 7f5b177fe700 1 -- 172.21.15.91:0/1326340830 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b24089570 con 0x7f5b3006b620 2023-11-24T06:09:20.401 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.395+0000 7f5b2e7fc700 1 --2- 172.21.15.91:0/1326340830 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f5b1c06af60 0x7f5b1c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.401 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.396+0000 7f5b2e7fc700 1 --2- 172.21.15.91:0/1326340830 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f5b1c06af60 0x7f5b1c06d420 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5b300cfc50 tx=0x7f5b20009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:20.402 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.400+0000 7f5b177fe700 1 -- 172.21.15.91:0/1326340830 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b240545c0 con 0x7f5b3006b620 2023-11-24T06:09:20.594 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.591+0000 7f5b35363700 1 -- 172.21.15.91:0/1326340830 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b30068670 con 0x7f5b3006b620 2023-11-24T06:09:20.595 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.592+0000 7f5b177fe700 1 -- 172.21.15.91:0/1326340830 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16485 (secure 0 0 0) 0x7f5b24087e60 con 0x7f5b3006b620 2023-11-24T06:09:20.595 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:20.596 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":34,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:09:00.042725+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-11-24T06:08:56.059841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-11-24T06:08:57.408988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:09:20.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.595+0000 7f5b35363700 1 -- 172.21.15.91:0/1326340830 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f5b1c06af60 msgr2=0x7f5b1c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:20.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.595+0000 7f5b35363700 1 --2- 172.21.15.91:0/1326340830 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f5b1c06af60 0x7f5b1c06d420 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5b300cfc50 tx=0x7f5b20009210 comp rx=0 tx=0).stop 2023-11-24T06:09:20.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.595+0000 7f5b35363700 1 -- 172.21.15.91:0/1326340830 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5b3006b620 msgr2=0x7f5b300cc810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:20.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.595+0000 7f5b35363700 1 --2- 172.21.15.91:0/1326340830 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5b3006b620 0x7f5b300cc810 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5b2400e930 tx=0x7f5b2400ec40 comp rx=0 tx=0).stop 2023-11-24T06:09:20.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.595+0000 7f5b35363700 1 -- 172.21.15.91:0/1326340830 shutdown_connections 2023-11-24T06:09:20.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.595+0000 7f5b35363700 1 --2- 172.21.15.91:0/1326340830 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f5b1c06af60 0x7f5b1c06d420 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.598 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.595+0000 7f5b35363700 1 --2- 172.21.15.91:0/1326340830 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f5b301478c0 0x7f5b300c9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.595+0000 7f5b35363700 1 --2- 172.21.15.91:0/1326340830 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f5b3006b620 0x7f5b300cc810 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.595+0000 7f5b35363700 1 --2- 172.21.15.91:0/1326340830 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f5b300b5ab0 0x7f5b300c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.595+0000 7f5b35363700 1 -- 172.21.15.91:0/1326340830 >> 172.21.15.91:0/1326340830 conn(0x7f5b300b2740 msgr2=0x7f5b300af120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:20.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.595+0000 7f5b35363700 1 -- 172.21.15.91:0/1326340830 shutdown_connections 2023-11-24T06:09:20.599 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.595+0000 7f5b35363700 1 -- 172.21.15.91:0/1326340830 wait complete. 2023-11-24T06:09:20.610 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-11-24T06:09:20.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.728+0000 7fd52c355700 1 Processor -- start 2023-11-24T06:09:20.731 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.728+0000 7fd52c355700 1 -- start start 2023-11-24T06:09:20.734 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.729+0000 7fd52c355700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd52415bf10 0x7fd524066ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.734 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.729+0000 7fd52c355700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd52406b7c0 0x7fd52406bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.734 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.729+0000 7fd52c355700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5240b10c0 con 0x7fd5240af980 2023-11-24T06:09:20.734 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.729+0000 7fd52c355700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd52406c190 con 0x7fd52406b7c0 2023-11-24T06:09:20.734 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.729+0000 7fd52c355700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5240af730 con 0x7fd52415bf10 2023-11-24T06:09:20.734 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.729+0000 7fd52a8f2700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd52406b7c0 0x7fd52406bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.734 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.729+0000 7fd52a8f2700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fd52406b7c0 0x7fd52406bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60230/0 (socket says 172.21.15.91:60230) 2023-11-24T06:09:20.734 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.729+0000 7fd52a8f2700 1 -- 172.21.15.91:0/4103987233 learned_addr learned my addr 172.21.15.91:0/4103987233 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:20.734 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.730+0000 7fd52a0f1700 1 --2- 172.21.15.91:0/4103987233 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd52415bf10 0x7fd524066ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.735 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.730+0000 7fd52a8f2700 1 -- 172.21.15.91:0/4103987233 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd52415bf10 msgr2=0x7fd524066ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:20.735 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.730+0000 7fd52a8f2700 1 --2- 172.21.15.91:0/4103987233 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd52415bf10 0x7fd524066ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.735 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.730+0000 7fd52a8f2700 1 -- 172.21.15.91:0/4103987233 >> v1:172.21.15.94:6789/0 conn(0x7fd5240af980 legacy=0x7fd5240b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:20.735 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.730+0000 7fd52a8f2700 1 -- 172.21.15.91:0/4103987233 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5240b1c60 con 0x7fd52406b7c0 2023-11-24T06:09:20.735 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.730+0000 7fd52a0f1700 1 --2- 172.21.15.91:0/4103987233 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd52415bf10 0x7fd524066ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:20.735 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.730+0000 7fd52a8f2700 1 --2- 172.21.15.91:0/4103987233 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd52406b7c0 0x7fd52406bc50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009e50 tx=0x7fd51c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95bb1fec09a6028 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:20.735 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.730+0000 7fd5290ef700 1 -- 172.21.15.91:0/4103987233 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c00e070 con 0x7fd52406b7c0 2023-11-24T06:09:20.735 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.730+0000 7fd5290ef700 1 -- 172.21.15.91:0/4103987233 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51c00b9c0 con 0x7fd52406b7c0 2023-11-24T06:09:20.735 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.730+0000 7fd5290ef700 1 -- 172.21.15.91:0/4103987233 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c00bd70 con 0x7fd52406b7c0 2023-11-24T06:09:20.735 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.730+0000 7fd52c355700 1 -- 172.21.15.91:0/4103987233 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd52406b7c0 msgr2=0x7fd52406bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:20.735 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.730+0000 7fd52c355700 1 --2- 172.21.15.91:0/4103987233 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd52406b7c0 0x7fd52406bc50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009e50 tx=0x7fd51c00b330 comp rx=0 tx=0).stop 2023-11-24T06:09:20.735 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.731+0000 7fd52c355700 1 -- 172.21.15.91:0/4103987233 shutdown_connections 2023-11-24T06:09:20.735 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.731+0000 7fd52c355700 1 --2- 172.21.15.91:0/4103987233 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd52406b7c0 0x7fd52406bc50 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.736 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.731+0000 7fd52c355700 1 --2- 172.21.15.91:0/4103987233 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd52415bf10 0x7fd524066ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.736 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.731+0000 7fd52c355700 1 -- 172.21.15.91:0/4103987233 >> 172.21.15.91:0/4103987233 conn(0x7fd5240b5070 msgr2=0x7fd5240b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:20.736 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.731+0000 7fd52c355700 1 -- 172.21.15.91:0/4103987233 shutdown_connections 2023-11-24T06:09:20.736 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.731+0000 7fd52c355700 1 -- 172.21.15.91:0/4103987233 wait complete. 2023-11-24T06:09:20.738 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.732+0000 7fd52c355700 1 Processor -- start 2023-11-24T06:09:20.738 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.732+0000 7fd52c355700 1 -- start start 2023-11-24T06:09:20.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.733+0000 7fd52c355700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd52406b7c0 0x7fd524151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.733+0000 7fd52c355700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5240af980 0x7fd524099810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.733+0000 7fd52c355700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd52415bf10 0x7fd524153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.733+0000 7fd52c355700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd524142560 con 0x7fd52415bf10 2023-11-24T06:09:20.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.733+0000 7fd52c355700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd524151670 con 0x7fd5240af980 2023-11-24T06:09:20.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.733+0000 7fd52c355700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5241418e0 con 0x7fd52406b7c0 2023-11-24T06:09:20.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.733+0000 7fd52a8f2700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd52415bf10 0x7fd524153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.733+0000 7fd52a8f2700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd52415bf10 0x7fd524153970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60236/0 (socket says 172.21.15.91:60236) 2023-11-24T06:09:20.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.733+0000 7fd52a8f2700 1 -- 172.21.15.91:0/3023665645 learned_addr learned my addr 172.21.15.91:0/3023665645 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:20.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.733+0000 7fd5298f0700 1 --2- 172.21.15.91:0/3023665645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5240af980 0x7fd524099810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.739 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.733+0000 7fd52a0f1700 1 --2- 172.21.15.91:0/3023665645 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd52406b7c0 0x7fd524151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.733+0000 7fd52a8f2700 1 -- 172.21.15.91:0/3023665645 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd52406b7c0 msgr2=0x7fd524151130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:20.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.733+0000 7fd52a8f2700 1 --2- 172.21.15.91:0/3023665645 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd52406b7c0 0x7fd524151130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.733+0000 7fd52a8f2700 1 -- 172.21.15.91:0/3023665645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5240af980 msgr2=0x7fd524099810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:20.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.733+0000 7fd52a8f2700 1 --2- 172.21.15.91:0/3023665645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5240af980 0x7fd524099810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.734+0000 7fd52a8f2700 1 -- 172.21.15.91:0/3023665645 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51c009b30 con 0x7fd52415bf10 2023-11-24T06:09:20.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.734+0000 7fd52a0f1700 1 --2- 172.21.15.91:0/3023665645 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd52406b7c0 0x7fd524151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:20.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.734+0000 7fd52a8f2700 1 --2- 172.21.15.91:0/3023665645 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd52415bf10 0x7fd524153970 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00b730 tx=0x7fd51c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:20.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.734+0000 7fd51affd700 1 -- 172.21.15.91:0/3023665645 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c0092e0 con 0x7fd52415bf10 2023-11-24T06:09:20.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.734+0000 7fd51affd700 1 -- 172.21.15.91:0/3023665645 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51c0099b0 con 0x7fd52415bf10 2023-11-24T06:09:20.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.734+0000 7fd52c355700 1 -- 172.21.15.91:0/3023665645 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd524141bc0 con 0x7fd52415bf10 2023-11-24T06:09:20.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.734+0000 7fd51affd700 1 -- 172.21.15.91:0/3023665645 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51c01c680 con 0x7fd52415bf10 2023-11-24T06:09:20.740 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.734+0000 7fd52c355700 1 -- 172.21.15.91:0/3023665645 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd524142180 con 0x7fd52415bf10 2023-11-24T06:09:20.741 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.735+0000 7fd52c355700 1 -- 172.21.15.91:0/3023665645 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd524068670 con 0x7fd52415bf10 2023-11-24T06:09:20.741 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.735+0000 7fd51affd700 1 -- 172.21.15.91:0/3023665645 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fd51c01c7e0 con 0x7fd52415bf10 2023-11-24T06:09:20.741 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.736+0000 7fd51affd700 1 --2- 172.21.15.91:0/3023665645 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd50806af60 0x7fd50806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:20.741 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.736+0000 7fd51affd700 1 -- 172.21.15.91:0/3023665645 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd51c08b590 con 0x7fd52415bf10 2023-11-24T06:09:20.741 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.736+0000 7fd52a0f1700 1 --2- 172.21.15.91:0/3023665645 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd50806af60 0x7fd50806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:20.741 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.736+0000 7fd52a0f1700 1 --2- 172.21.15.91:0/3023665645 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd50806af60 0x7fd50806d420 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd514008910 tx=0x7fd514008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:20.743 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.740+0000 7fd51affd700 1 -- 172.21.15.91:0/3023665645 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd51c056550 con 0x7fd52415bf10 2023-11-24T06:09:20.977 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.974+0000 7fd52c355700 1 -- 172.21.15.91:0/3023665645 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd5240b5c20 con 0x7fd52415bf10 2023-11-24T06:09:20.977 INFO:tasks.ceph.mon.a.smithi091.stderr:2023-11-24T06:09:20.974+0000 7f8e5603a700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-11-24T06:09:20.980 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.977+0000 7fd51affd700 1 -- 172.21.15.91:0/3023665645 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd51c059b70 con 0x7fd52415bf10 2023-11-24T06:09:20.983 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.980+0000 7fd52c355700 1 -- 172.21.15.91:0/3023665645 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd50806af60 msgr2=0x7fd50806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:20.983 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.980+0000 7fd52c355700 1 --2- 172.21.15.91:0/3023665645 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd50806af60 0x7fd50806d420 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd514008910 tx=0x7fd514008040 comp rx=0 tx=0).stop 2023-11-24T06:09:20.983 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.980+0000 7fd52c355700 1 -- 172.21.15.91:0/3023665645 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd52415bf10 msgr2=0x7fd524153970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:20.983 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.980+0000 7fd52c355700 1 --2- 172.21.15.91:0/3023665645 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd52415bf10 0x7fd524153970 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00b730 tx=0x7fd51c004ad0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.984 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.981+0000 7fd52c355700 1 -- 172.21.15.91:0/3023665645 shutdown_connections 2023-11-24T06:09:20.984 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.981+0000 7fd52c355700 1 --2- 172.21.15.91:0/3023665645 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd50806af60 0x7fd50806d420 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.984 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.981+0000 7fd52c355700 1 --2- 172.21.15.91:0/3023665645 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd52415bf10 0x7fd524153970 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.984 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.981+0000 7fd52c355700 1 --2- 172.21.15.91:0/3023665645 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd52406b7c0 0x7fd524151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.984 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.981+0000 7fd52c355700 1 --2- 172.21.15.91:0/3023665645 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd5240af980 0x7fd524099810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:20.984 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.981+0000 7fd52c355700 1 -- 172.21.15.91:0/3023665645 >> 172.21.15.91:0/3023665645 conn(0x7fd5240b5070 msgr2=0x7fd52415ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:20.985 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.981+0000 7fd52c355700 1 -- 172.21.15.91:0/3023665645 shutdown_connections 2023-11-24T06:09:20.985 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:20.981+0000 7fd52c355700 1 -- 172.21.15.91:0/3023665645 wait complete. 2023-11-24T06:09:20.986 INFO:teuthology.orchestra.run.smithi091.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-11-24T06:09:20.987 INFO:tasks.ceph.mds.a.smithi091.stderr: -7426> 2023-11-24T06:09:00.976+0000 7f4039daf700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-11-24T06:09:21.001 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-11-24T06:09:21.026 INFO:tasks.ceph.mds.a.smithi091.stdout:starting mds.a at 2023-11-24T06:09:21.124 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.121+0000 7ff22464c700 1 Processor -- start 2023-11-24T06:09:21.124 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.121+0000 7ff22464c700 1 -- start start 2023-11-24T06:09:21.127 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.122+0000 7ff22464c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff21c143ae0 0x7ff21c0b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:21.127 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.122+0000 7ff22464c700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff21c143ec0 con 0x7ff21c1440f0 2023-11-24T06:09:21.127 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.122+0000 7ff22464c700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff21c0b3ae0 con 0x7ff21c0722c0 2023-11-24T06:09:21.127 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.122+0000 7ff22464c700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff21c0b40a0 con 0x7ff21c143ae0 2023-11-24T06:09:21.127 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.123+0000 7ff2223e8700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff21c143ae0 0x7ff21c0b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:21.127 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.123+0000 7ff221be7700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7ff21c1440f0 0x7ff21c071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55186/0 (socket says 172.21.15.91:55186) 2023-11-24T06:09:21.127 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.123+0000 7ff2223e8700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff21c143ae0 0x7ff21c0b51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60252/0 (socket says 172.21.15.91:60252) 2023-11-24T06:09:21.128 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.123+0000 7ff221be7700 1 -- 172.21.15.91:0/1853018279 learned_addr learned my addr 172.21.15.91:0/1853018279 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:21.128 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.123+0000 7ff2223e8700 1 -- 172.21.15.91:0/1853018279 >> v1:172.21.15.94:6789/0 conn(0x7ff21c0722c0 legacy=0x7ff21c06e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:21.128 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.123+0000 7ff2223e8700 1 -- 172.21.15.91:0/1853018279 >> v1:172.21.15.91:6789/0 conn(0x7ff21c1440f0 legacy=0x7ff21c071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:21.128 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.123+0000 7ff2223e8700 1 -- 172.21.15.91:0/1853018279 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21c0b3dc0 con 0x7ff21c143ae0 2023-11-24T06:09:21.128 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.123+0000 7ff2213e6700 1 -- 172.21.15.91:0/1853018279 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856484024 0 0) 0x7ff21c143ec0 con 0x7ff21c1440f0 2023-11-24T06:09:21.128 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.123+0000 7ff2223e8700 1 --2- 172.21.15.91:0/1853018279 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff21c143ae0 0x7ff21c0b51b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff20c009840 tx=0x7ff20c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc76a2092f5fbe0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:21.128 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.123+0000 7ff2213e6700 1 -- 172.21.15.91:0/1853018279 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c004230 con 0x7ff21c143ae0 2023-11-24T06:09:21.128 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.123+0000 7ff2213e6700 1 -- 172.21.15.91:0/1853018279 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20c004390 con 0x7ff21c143ae0 2023-11-24T06:09:21.128 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.123+0000 7ff2213e6700 1 -- 172.21.15.91:0/1853018279 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c0036a0 con 0x7ff21c143ae0 2023-11-24T06:09:21.128 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.124+0000 7ff22464c700 1 -- 172.21.15.91:0/1853018279 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff21c143ae0 msgr2=0x7ff21c0b51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:21.129 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.124+0000 7ff22464c700 1 --2- 172.21.15.91:0/1853018279 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff21c143ae0 0x7ff21c0b51b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff20c009840 tx=0x7ff20c009b50 comp rx=0 tx=0).stop 2023-11-24T06:09:21.129 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.124+0000 7ff22464c700 1 -- 172.21.15.91:0/1853018279 shutdown_connections 2023-11-24T06:09:21.129 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.124+0000 7ff22464c700 1 --2- 172.21.15.91:0/1853018279 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff21c143ae0 0x7ff21c0b51b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:21.129 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.124+0000 7ff22464c700 1 -- 172.21.15.91:0/1853018279 >> 172.21.15.91:0/1853018279 conn(0x7ff21c0b2740 msgr2=0x7ff21c0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:21.129 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.124+0000 7ff22464c700 1 -- 172.21.15.91:0/1853018279 shutdown_connections 2023-11-24T06:09:21.129 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.124+0000 7ff22464c700 1 -- 172.21.15.91:0/1853018279 wait complete. 2023-11-24T06:09:21.132 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.125+0000 7ff22464c700 1 Processor -- start 2023-11-24T06:09:21.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.125+0000 7ff22464c700 1 -- start start 2023-11-24T06:09:21.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff22464c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff21c0722c0 0x7ff21c15d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:21.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff22464c700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff21c143ae0 0x7ff21c155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:21.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff22464c700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c1440f0 0x7ff21c15bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:21.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff22464c700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c15c8b0 con 0x7ff21c0722c0 2023-11-24T06:09:21.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff22464c700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c15cb90 con 0x7ff21c1440f0 2023-11-24T06:09:21.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff22464c700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c15ce70 con 0x7ff21c143ae0 2023-11-24T06:09:21.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff2223e8700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff21c0722c0 0x7ff21c15d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:21.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff2223e8700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff21c0722c0 0x7ff21c15d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60258/0 (socket says 172.21.15.91:60258) 2023-11-24T06:09:21.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff2223e8700 1 -- 172.21.15.91:0/485052842 learned_addr learned my addr 172.21.15.91:0/485052842 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:21.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff222be9700 1 --2- 172.21.15.91:0/485052842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c1440f0 0x7ff21c15bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:21.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff221be7700 1 --2- 172.21.15.91:0/485052842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff21c143ae0 0x7ff21c155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:21.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff2223e8700 1 -- 172.21.15.91:0/485052842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff21c143ae0 msgr2=0x7ff21c155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:21.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff2223e8700 1 --2- 172.21.15.91:0/485052842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff21c143ae0 0x7ff21c155cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:21.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff2223e8700 1 -- 172.21.15.91:0/485052842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c1440f0 msgr2=0x7ff21c15bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:21.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff2223e8700 1 --2- 172.21.15.91:0/485052842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c1440f0 0x7ff21c15bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:21.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff2223e8700 1 -- 172.21.15.91:0/485052842 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c0094a0 con 0x7ff21c0722c0 2023-11-24T06:09:21.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff222be9700 1 --2- 172.21.15.91:0/485052842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c1440f0 0x7ff21c15bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:21.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff221be7700 1 --2- 172.21.15.91:0/485052842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff21c143ae0 0x7ff21c155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:21.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff2223e8700 1 --2- 172.21.15.91:0/485052842 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff21c0722c0 0x7ff21c15d950 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0090b0 tx=0x7ff20c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:21.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff212ffd700 1 -- 172.21.15.91:0/485052842 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c011830 con 0x7ff21c0722c0 2023-11-24T06:09:21.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff212ffd700 1 -- 172.21.15.91:0/485052842 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20c01e070 con 0x7ff21c0722c0 2023-11-24T06:09:21.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.126+0000 7ff22464c700 1 -- 172.21.15.91:0/485052842 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff21c15d150 con 0x7ff21c0722c0 2023-11-24T06:09:21.135 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.127+0000 7ff212ffd700 1 -- 172.21.15.91:0/485052842 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c011bb0 con 0x7ff21c0722c0 2023-11-24T06:09:21.135 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.127+0000 7ff22464c700 1 -- 172.21.15.91:0/485052842 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff21c156210 con 0x7ff21c0722c0 2023-11-24T06:09:21.135 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.127+0000 7ff22464c700 1 -- 172.21.15.91:0/485052842 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff21c15d7a0 con 0x7ff21c0722c0 2023-11-24T06:09:21.135 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.128+0000 7ff212ffd700 1 -- 172.21.15.91:0/485052842 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7ff20c011d10 con 0x7ff21c0722c0 2023-11-24T06:09:21.135 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.128+0000 7ff212ffd700 1 --2- 172.21.15.91:0/485052842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff20006af60 0x7ff20006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:21.135 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.128+0000 7ff212ffd700 1 -- 172.21.15.91:0/485052842 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff20c08aa10 con 0x7ff21c0722c0 2023-11-24T06:09:21.135 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.128+0000 7ff221be7700 1 --2- 172.21.15.91:0/485052842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff20006af60 0x7ff20006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:21.135 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.129+0000 7ff221be7700 1 --2- 172.21.15.91:0/485052842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff20006af60 0x7ff20006d420 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff21c156e70 tx=0x7ff218009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:21.136 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.133+0000 7ff212ffd700 1 -- 172.21.15.91:0/485052842 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff20c055b00 con 0x7ff21c0722c0 2023-11-24T06:09:21.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.365+0000 7ff22464c700 1 -- 172.21.15.91:0/485052842 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff21c075480 con 0x7ff21c0722c0 2023-11-24T06:09:21.984 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.982+0000 7ff212ffd700 1 -- 172.21.15.91:0/485052842 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff20c0200a0 con 0x7ff21c0722c0 2023-11-24T06:09:21.987 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.984+0000 7ff22464c700 1 -- 172.21.15.91:0/485052842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff20006af60 msgr2=0x7ff20006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:21.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.984+0000 7ff22464c700 1 --2- 172.21.15.91:0/485052842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff20006af60 0x7ff20006d420 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff21c156e70 tx=0x7ff218009040 comp rx=0 tx=0).stop 2023-11-24T06:09:21.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.984+0000 7ff22464c700 1 -- 172.21.15.91:0/485052842 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff21c0722c0 msgr2=0x7ff21c15d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:21.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.984+0000 7ff22464c700 1 --2- 172.21.15.91:0/485052842 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff21c0722c0 0x7ff21c15d950 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0090b0 tx=0x7ff20c004750 comp rx=0 tx=0).stop 2023-11-24T06:09:21.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.984+0000 7ff22464c700 1 -- 172.21.15.91:0/485052842 shutdown_connections 2023-11-24T06:09:21.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.984+0000 7ff22464c700 1 --2- 172.21.15.91:0/485052842 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff20006af60 0x7ff20006d420 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:21.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.984+0000 7ff22464c700 1 --2- 172.21.15.91:0/485052842 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff21c1440f0 0x7ff21c15bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:21.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.984+0000 7ff22464c700 1 --2- 172.21.15.91:0/485052842 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff21c0722c0 0x7ff21c15d950 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:21.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.984+0000 7ff22464c700 1 --2- 172.21.15.91:0/485052842 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff21c143ae0 0x7ff21c155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:21.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.984+0000 7ff22464c700 1 -- 172.21.15.91:0/485052842 >> 172.21.15.91:0/485052842 conn(0x7ff21c0b2740 msgr2=0x7ff21c0cf7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:21.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.984+0000 7ff22464c700 1 -- 172.21.15.91:0/485052842 shutdown_connections 2023-11-24T06:09:21.988 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:21.984+0000 7ff22464c700 1 -- 172.21.15.91:0/485052842 wait complete. 2023-11-24T06:09:22.001 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-11-24T06:09:22.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.172+0000 7f0e91e3d700 1 Processor -- start 2023-11-24T06:09:22.175 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.172+0000 7f0e91e3d700 1 -- start start 2023-11-24T06:09:22.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.172+0000 7f0e91e3d700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f0e8c15bfe0 0x7f0e8c0cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:22.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.172+0000 7f0e91e3d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e8c0ce430 0x7f0e8c0ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:22.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.172+0000 7f0e91e3d700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f0e8c0cedf0 0x7f0e8c0cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:22.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.172+0000 7f0e91e3d700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8c06f4b0 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.172+0000 7f0e91e3d700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e8c0cf260 con 0x7f0e8c15bfe0 2023-11-24T06:09:22.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.172+0000 7f0e91e3d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c15c3c0 con 0x7f0e8c0ce430 2023-11-24T06:09:22.177 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.173+0000 7f0e8bfff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f0e8c0cedf0 0x7f0e8c0cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:22.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.173+0000 7f0e8bfff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f0e8c0cedf0 0x7f0e8c0cf7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60272/0 (socket says 172.21.15.91:60272) 2023-11-24T06:09:22.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.173+0000 7f0e8bfff700 1 -- 172.21.15.91:0/1103628239 learned_addr learned my addr 172.21.15.91:0/1103628239 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:22.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.173+0000 7f0e8bfff700 1 -- 172.21.15.91:0/1103628239 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e8c0ce430 msgr2=0x7f0e8c0ce8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:22.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.173+0000 7f0e8b7fe700 1 --2- 172.21.15.91:0/1103628239 >> v2:172.21.15.94:3300/0 conn(0x7f0e8c15bfe0 0x7f0e8c0cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:22.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.173+0000 7f0e8affd700 1 --2- 172.21.15.91:0/1103628239 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e8c0ce430 0x7f0e8c0ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:22.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.173+0000 7f0e8bfff700 1 --2- 172.21.15.91:0/1103628239 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e8c0ce430 0x7f0e8c0ce8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:22.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.173+0000 7f0e8bfff700 1 -- 172.21.15.91:0/1103628239 >> v2:172.21.15.94:3300/0 conn(0x7f0e8c15bfe0 msgr2=0x7f0e8c0cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:22.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.173+0000 7f0e8bfff700 1 --2- 172.21.15.91:0/1103628239 >> v2:172.21.15.94:3300/0 conn(0x7f0e8c15bfe0 0x7f0e8c0cd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:22.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.173+0000 7f0e8bfff700 1 -- 172.21.15.91:0/1103628239 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8c0cd760 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.173+0000 7f0e8bfff700 1 --2- 172.21.15.91:0/1103628239 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e8c0cedf0 0x7f0e8c0cf7a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c06ebf0 tx=0x7f0e8400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b57dba75cce9fce server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:22.178 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.173+0000 7f0e8a7fc700 1 -- 172.21.15.91:0/1103628239 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8400e070 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.179 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.173+0000 7f0e8a7fc700 1 -- 172.21.15.91:0/1103628239 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e8400b9c0 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.179 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.173+0000 7f0e8a7fc700 1 -- 172.21.15.91:0/1103628239 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8400bd70 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.179 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.174+0000 7f0e91e3d700 1 -- 172.21.15.91:0/1103628239 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e8c0cedf0 msgr2=0x7f0e8c0cf7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:22.179 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.174+0000 7f0e91e3d700 1 --2- 172.21.15.91:0/1103628239 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e8c0cedf0 0x7f0e8c0cf7a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c06ebf0 tx=0x7f0e8400b330 comp rx=0 tx=0).stop 2023-11-24T06:09:22.179 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.174+0000 7f0e91e3d700 1 -- 172.21.15.91:0/1103628239 shutdown_connections 2023-11-24T06:09:22.179 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.174+0000 7f0e91e3d700 1 --2- 172.21.15.91:0/1103628239 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e8c0cedf0 0x7f0e8c0cf7a0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:22.179 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.174+0000 7f0e91e3d700 1 --2- 172.21.15.91:0/1103628239 >> v2:172.21.15.94:3300/0 conn(0x7f0e8c15bfe0 0x7f0e8c0cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:22.179 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.174+0000 7f0e91e3d700 1 --2- 172.21.15.91:0/1103628239 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e8c0ce430 0x7f0e8c0ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:22.179 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.174+0000 7f0e91e3d700 1 -- 172.21.15.91:0/1103628239 >> 172.21.15.91:0/1103628239 conn(0x7f0e8c0b3700 msgr2=0x7f0e8c0b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:22.179 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.174+0000 7f0e91e3d700 1 -- 172.21.15.91:0/1103628239 shutdown_connections 2023-11-24T06:09:22.180 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.174+0000 7f0e91e3d700 1 -- 172.21.15.91:0/1103628239 wait complete. 2023-11-24T06:09:22.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.175+0000 7f0e91e3d700 1 Processor -- start 2023-11-24T06:09:22.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.175+0000 7f0e91e3d700 1 -- start start 2023-11-24T06:09:22.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e91e3d700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e8c0ce430 0x7f0e8c1591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:22.182 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e91e3d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e8c0cedf0 0x7f0e8c1f2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:22.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e91e3d700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e8c15bfe0 0x7f0e8c0aea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:22.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e91e3d700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c159720 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e91e3d700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c0ca5d0 con 0x7f0e8c15bfe0 2023-11-24T06:09:22.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e91e3d700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8c0cbe00 con 0x7f0e8c0ce430 2023-11-24T06:09:22.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e8affd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e8c0cedf0 0x7f0e8c1f2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:22.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e8affd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e8c0cedf0 0x7f0e8c1f2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60276/0 (socket says 172.21.15.91:60276) 2023-11-24T06:09:22.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e8affd700 1 -- 172.21.15.91:0/2635353493 learned_addr learned my addr 172.21.15.91:0/2635353493 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:22.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e8b7fe700 1 --2- 172.21.15.91:0/2635353493 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e8c0ce430 0x7f0e8c1591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:22.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e8bfff700 1 --2- 172.21.15.91:0/2635353493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e8c15bfe0 0x7f0e8c0aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:22.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e8affd700 1 -- 172.21.15.91:0/2635353493 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e8c0ce430 msgr2=0x7f0e8c1591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:22.183 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e8affd700 1 --2- 172.21.15.91:0/2635353493 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e8c0ce430 0x7f0e8c1591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:22.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e8affd700 1 -- 172.21.15.91:0/2635353493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e8c15bfe0 msgr2=0x7f0e8c0aea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:22.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e8affd700 1 --2- 172.21.15.91:0/2635353493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e8c15bfe0 0x7f0e8c0aea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:22.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e8affd700 1 -- 172.21.15.91:0/2635353493 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e84009b30 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e8bfff700 1 --2- 172.21.15.91:0/2635353493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e8c15bfe0 0x7f0e8c0aea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:22.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.176+0000 7f0e8affd700 1 --2- 172.21.15.91:0/2635353493 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e8c0cedf0 0x7f0e8c1f2010 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400c7f0 tx=0x7f0e7400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:22.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.177+0000 7f0e83fff700 1 -- 172.21.15.91:0/2635353493 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e74011800 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.177+0000 7f0e83fff700 1 -- 172.21.15.91:0/2635353493 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e74011960 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.177+0000 7f0e91e3d700 1 -- 172.21.15.91:0/2635353493 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8c1f2550 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.177+0000 7f0e83fff700 1 -- 172.21.15.91:0/2635353493 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e74011cf0 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.177+0000 7f0e91e3d700 1 -- 172.21.15.91:0/2635353493 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e8c0c9ca0 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.184 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.178+0000 7f0e91e3d700 1 -- 172.21.15.91:0/2635353493 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e8c06bc40 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.185 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.178+0000 7f0e83fff700 1 -- 172.21.15.91:0/2635353493 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f0e74018360 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.185 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.179+0000 7f0e83fff700 1 --2- 172.21.15.91:0/2635353493 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0e6c06af60 0x7f0e6c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:22.185 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.179+0000 7f0e8b7fe700 1 --2- 172.21.15.91:0/2635353493 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0e6c06af60 0x7f0e6c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:22.185 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.179+0000 7f0e83fff700 1 -- 172.21.15.91:0/2635353493 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0e74089690 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.185 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.179+0000 7f0e8b7fe700 1 --2- 172.21.15.91:0/2635353493 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0e6c06af60 0x7f0e6c06d420 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00a910 tx=0x7f0e7c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:22.186 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.183+0000 7f0e83fff700 1 -- 172.21.15.91:0/2635353493 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e740545e0 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.376 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.374+0000 7f0e91e3d700 1 -- 172.21.15.91:0/2635353493 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0e8c068670 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.985 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.982+0000 7f0e83fff700 1 -- 172.21.15.91:0/2635353493 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v36) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0e74057c00 con 0x7f0e8c0cedf0 2023-11-24T06:09:22.989 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.985+0000 7f0e91e3d700 1 -- 172.21.15.91:0/2635353493 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0e6c06af60 msgr2=0x7f0e6c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:22.989 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.985+0000 7f0e91e3d700 1 --2- 172.21.15.91:0/2635353493 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0e6c06af60 0x7f0e6c06d420 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00a910 tx=0x7f0e7c008040 comp rx=0 tx=0).stop 2023-11-24T06:09:22.989 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.985+0000 7f0e91e3d700 1 -- 172.21.15.91:0/2635353493 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e8c0cedf0 msgr2=0x7f0e8c1f2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:22.989 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.985+0000 7f0e91e3d700 1 --2- 172.21.15.91:0/2635353493 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e8c0cedf0 0x7f0e8c1f2010 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0e7400c7f0 tx=0x7f0e7400cbb0 comp rx=0 tx=0).stop 2023-11-24T06:09:22.989 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.985+0000 7f0e91e3d700 1 -- 172.21.15.91:0/2635353493 shutdown_connections 2023-11-24T06:09:22.989 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.985+0000 7f0e91e3d700 1 --2- 172.21.15.91:0/2635353493 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f0e6c06af60 0x7f0e6c06d420 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:22.989 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.985+0000 7f0e91e3d700 1 --2- 172.21.15.91:0/2635353493 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f0e8c15bfe0 0x7f0e8c0aea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:22.990 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.985+0000 7f0e91e3d700 1 --2- 172.21.15.91:0/2635353493 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f0e8c0ce430 0x7f0e8c1591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:22.990 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.985+0000 7f0e91e3d700 1 --2- 172.21.15.91:0/2635353493 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f0e8c0cedf0 0x7f0e8c1f2010 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:22.990 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.985+0000 7f0e91e3d700 1 -- 172.21.15.91:0/2635353493 >> 172.21.15.91:0/2635353493 conn(0x7f0e8c0b3700 msgr2=0x7f0e8c0b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:22.990 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.985+0000 7f0e91e3d700 1 -- 172.21.15.91:0/2635353493 shutdown_connections 2023-11-24T06:09:22.990 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:22.985+0000 7f0e91e3d700 1 -- 172.21.15.91:0/2635353493 wait complete. 2023-11-24T06:09:22.990 INFO:teuthology.orchestra.run.smithi091.stderr:pool 'cephfs_metadata' removed 2023-11-24T06:09:23.002 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-11-24T06:09:23.130 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.126+0000 7f4bf1ba7700 1 Processor -- start 2023-11-24T06:09:23.131 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.126+0000 7f4bf1ba7700 1 -- start start 2023-11-24T06:09:23.132 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.127+0000 7f4bf1ba7700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bec1447a0 0x7f4bec071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:23.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.127+0000 7f4bf1ba7700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4bec06f070 0x7f4bec0720d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:23.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.127+0000 7f4bf1ba7700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec06dbd0 con 0x7f4bec072ab0 2023-11-24T06:09:23.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.127+0000 7f4bf1ba7700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bec15b700 con 0x7f4bec06f070 2023-11-24T06:09:23.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.127+0000 7f4bf1ba7700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec15b9e0 con 0x7f4bec1447a0 2023-11-24T06:09:23.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.127+0000 7f4bebfff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4bec06f070 0x7f4bec0720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:23.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.127+0000 7f4beaffd700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f4bec072ab0 0x7f4bec15e750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55218/0 (socket says 172.21.15.91:55218) 2023-11-24T06:09:23.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.127+0000 7f4beaffd700 1 -- 172.21.15.91:0/2377734857 learned_addr learned my addr 172.21.15.91:0/2377734857 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:23.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.128+0000 7f4bebfff700 1 -- 172.21.15.91:0/2377734857 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bec1447a0 msgr2=0x7f4bec071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:23.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.128+0000 7f4bebfff700 1 --2- 172.21.15.91:0/2377734857 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bec1447a0 0x7f4bec071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:23.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.128+0000 7f4bebfff700 1 -- 172.21.15.91:0/2377734857 >> v1:172.21.15.91:6789/0 conn(0x7f4bec072ab0 legacy=0x7f4bec15e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:23.133 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.128+0000 7f4bebfff700 1 -- 172.21.15.91:0/2377734857 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bec15bcc0 con 0x7f4bec06f070 2023-11-24T06:09:23.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.128+0000 7f4bebfff700 1 --2- 172.21.15.91:0/2377734857 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4bec06f070 0x7f4bec0720d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4be4009d90 tx=0x7f4be400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfec7a0abcd25e5c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:23.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.128+0000 7f4bea7fc700 1 -- 172.21.15.91:0/2377734857 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be400e070 con 0x7f4bec06f070 2023-11-24T06:09:23.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.128+0000 7f4bea7fc700 1 -- 172.21.15.91:0/2377734857 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be400bbd0 con 0x7f4bec06f070 2023-11-24T06:09:23.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.128+0000 7f4bea7fc700 1 -- 172.21.15.91:0/2377734857 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be4012430 con 0x7f4bec06f070 2023-11-24T06:09:23.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.128+0000 7f4bf1ba7700 1 -- 172.21.15.91:0/2377734857 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4bec06f070 msgr2=0x7f4bec0720d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:23.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.128+0000 7f4bf1ba7700 1 --2- 172.21.15.91:0/2377734857 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4bec06f070 0x7f4bec0720d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4be4009d90 tx=0x7f4be400b330 comp rx=0 tx=0).stop 2023-11-24T06:09:23.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.129+0000 7f4bf1ba7700 1 -- 172.21.15.91:0/2377734857 shutdown_connections 2023-11-24T06:09:23.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.129+0000 7f4bf1ba7700 1 --2- 172.21.15.91:0/2377734857 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4bec06f070 0x7f4bec0720d0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:23.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.129+0000 7f4bf1ba7700 1 --2- 172.21.15.91:0/2377734857 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bec1447a0 0x7f4bec071b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:23.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.129+0000 7f4bf1ba7700 1 -- 172.21.15.91:0/2377734857 >> 172.21.15.91:0/2377734857 conn(0x7f4bec0b2760 msgr2=0x7f4bec0b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:23.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.129+0000 7f4bf1ba7700 1 -- 172.21.15.91:0/2377734857 shutdown_connections 2023-11-24T06:09:23.134 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.129+0000 7f4bf1ba7700 1 -- 172.21.15.91:0/2377734857 wait complete. 2023-11-24T06:09:23.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.130+0000 7f4bf1ba7700 1 Processor -- start 2023-11-24T06:09:23.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.130+0000 7f4bf1ba7700 1 -- start start 2023-11-24T06:09:23.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4bf1ba7700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bec06f070 0x7f4bec06b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:23.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4bf1ba7700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4bec072ab0 0x7f4bec15a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:23.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4bf1ba7700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bec1447a0 0x7f4bec15a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:23.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4bf1ba7700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec139410 con 0x7f4bec072ab0 2023-11-24T06:09:23.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4bf1ba7700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec1f72d0 con 0x7f4bec1447a0 2023-11-24T06:09:23.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4bf1ba7700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec1f75b0 con 0x7f4bec06f070 2023-11-24T06:09:23.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4beaffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4bec072ab0 0x7f4bec15a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:23.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4beaffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4bec072ab0 0x7f4bec15a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60290/0 (socket says 172.21.15.91:60290) 2023-11-24T06:09:23.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4beaffd700 1 -- 172.21.15.91:0/920430832 learned_addr learned my addr 172.21.15.91:0/920430832 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:23.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4bebfff700 1 --2- 172.21.15.91:0/920430832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bec1447a0 0x7f4bec15a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:23.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4beb7fe700 1 --2- 172.21.15.91:0/920430832 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bec06f070 0x7f4bec06b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:23.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4beaffd700 1 -- 172.21.15.91:0/920430832 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bec06f070 msgr2=0x7f4bec06b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:23.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4beaffd700 1 --2- 172.21.15.91:0/920430832 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bec06f070 0x7f4bec06b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:23.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4beaffd700 1 -- 172.21.15.91:0/920430832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bec1447a0 msgr2=0x7f4bec15a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:23.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4beaffd700 1 --2- 172.21.15.91:0/920430832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bec1447a0 0x7f4bec15a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:23.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4beaffd700 1 -- 172.21.15.91:0/920430832 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be4009a70 con 0x7f4bec072ab0 2023-11-24T06:09:23.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4beaffd700 1 --2- 172.21.15.91:0/920430832 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4bec072ab0 0x7f4bec15a260 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00c520 tx=0x7f4bdc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:23.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4bd3fff700 1 -- 172.21.15.91:0/920430832 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc005d70 con 0x7f4bec072ab0 2023-11-24T06:09:23.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4bd3fff700 1 -- 172.21.15.91:0/920430832 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bdc0070d0 con 0x7f4bec072ab0 2023-11-24T06:09:23.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4bf1ba7700 1 -- 172.21.15.91:0/920430832 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bec1f7890 con 0x7f4bec072ab0 2023-11-24T06:09:23.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4bd3fff700 1 -- 172.21.15.91:0/920430832 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc007440 con 0x7f4bec072ab0 2023-11-24T06:09:23.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.131+0000 7f4bf1ba7700 1 -- 172.21.15.91:0/920430832 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bec1f8090 con 0x7f4bec072ab0 2023-11-24T06:09:23.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.132+0000 7f4bf1ba7700 1 -- 172.21.15.91:0/920430832 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bec068670 con 0x7f4bec072ab0 2023-11-24T06:09:23.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.133+0000 7f4bd3fff700 1 -- 172.21.15.91:0/920430832 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f4bdc0075a0 con 0x7f4bec072ab0 2023-11-24T06:09:23.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.133+0000 7f4bd3fff700 1 --2- 172.21.15.91:0/920430832 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4bd806af60 0x7f4bd806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:23.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.133+0000 7f4bd3fff700 1 -- 172.21.15.91:0/920430832 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f4bdc088910 con 0x7f4bec072ab0 2023-11-24T06:09:23.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.133+0000 7f4beb7fe700 1 --2- 172.21.15.91:0/920430832 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4bd806af60 0x7f4bd806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:23.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.134+0000 7f4beb7fe700 1 --2- 172.21.15.91:0/920430832 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4bd806af60 0x7f4bd806d420 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4be0006fd0 tx=0x7f4be0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:23.141 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.138+0000 7f4bd3fff700 1 -- 172.21.15.91:0/920430832 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4bdc053b60 con 0x7f4bec072ab0 2023-11-24T06:09:23.339 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.335+0000 7f4bf1ba7700 1 -- 172.21.15.91:0/920430832 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4bec0b1ac0 con 0x7f4bec072ab0 2023-11-24T06:09:23.990 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.986+0000 7f4bd3fff700 1 -- 172.21.15.91:0/920430832 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v37) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4bdc057180 con 0x7f4bec072ab0 2023-11-24T06:09:23.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.990+0000 7f4bf1ba7700 1 -- 172.21.15.91:0/920430832 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4bd806af60 msgr2=0x7f4bd806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:23.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.990+0000 7f4bf1ba7700 1 --2- 172.21.15.91:0/920430832 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4bd806af60 0x7f4bd806d420 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4be0006fd0 tx=0x7f4be0008040 comp rx=0 tx=0).stop 2023-11-24T06:09:23.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.990+0000 7f4bf1ba7700 1 -- 172.21.15.91:0/920430832 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4bec072ab0 msgr2=0x7f4bec15a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:23.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.990+0000 7f4bf1ba7700 1 --2- 172.21.15.91:0/920430832 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4bec072ab0 0x7f4bec15a260 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00c520 tx=0x7f4bdc00c8e0 comp rx=0 tx=0).stop 2023-11-24T06:09:23.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.990+0000 7f4bf1ba7700 1 -- 172.21.15.91:0/920430832 shutdown_connections 2023-11-24T06:09:23.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.990+0000 7f4bf1ba7700 1 --2- 172.21.15.91:0/920430832 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f4bd806af60 0x7f4bd806d420 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:23.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.990+0000 7f4bf1ba7700 1 --2- 172.21.15.91:0/920430832 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4bec1447a0 0x7f4bec15a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:23.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.990+0000 7f4bf1ba7700 1 --2- 172.21.15.91:0/920430832 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f4bec06f070 0x7f4bec06b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:23.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.990+0000 7f4bf1ba7700 1 --2- 172.21.15.91:0/920430832 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4bec072ab0 0x7f4bec15a260 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:23.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.990+0000 7f4bf1ba7700 1 -- 172.21.15.91:0/920430832 >> 172.21.15.91:0/920430832 conn(0x7f4bec0b2760 msgr2=0x7f4bec0ccad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:23.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.990+0000 7f4bf1ba7700 1 -- 172.21.15.91:0/920430832 shutdown_connections 2023-11-24T06:09:23.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:23.990+0000 7f4bf1ba7700 1 -- 172.21.15.91:0/920430832 wait complete. 2023-11-24T06:09:23.996 INFO:teuthology.orchestra.run.smithi091.stderr:pool 'cephfs_data' removed 2023-11-24T06:09:24.008 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-11-24T06:09:24.132 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.128+0000 7fd40be83700 1 Processor -- start 2023-11-24T06:09:24.132 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.128+0000 7fd40be83700 1 -- start start 2023-11-24T06:09:24.136 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.129+0000 7fd40be83700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fd40415bde0 0x7fd40415c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:24.136 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.129+0000 7fd40be83700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4040ce5c0 0x7fd4040cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:24.136 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.129+0000 7fd40be83700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4040d0a90 con 0x7fd4040cf2b0 2023-11-24T06:09:24.136 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.129+0000 7fd40be83700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4040cefb0 con 0x7fd40415bde0 2023-11-24T06:09:24.136 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.129+0000 7fd40be83700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40406ee10 con 0x7fd4040ce5c0 2023-11-24T06:09:24.136 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.129+0000 7fd40941e700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fd4040cf2b0 0x7fd4040d01b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55232/0 (socket says 172.21.15.91:55232) 2023-11-24T06:09:24.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.129+0000 7fd40941e700 1 -- 172.21.15.91:0/4043810615 learned_addr learned my addr 172.21.15.91:0/4043810615 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:24.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd40a420700 1 --2- 172.21.15.91:0/4043810615 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4040ce5c0 0x7fd4040cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:24.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd408c1d700 1 -- 172.21.15.91:0/4043810615 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105229755 0 0) 0x7fd4040d0a90 con 0x7fd4040cf2b0 2023-11-24T06:09:24.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd409c1f700 1 --2- 172.21.15.91:0/4043810615 >> v2:172.21.15.94:3300/0 conn(0x7fd40415bde0 0x7fd40415c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:24.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd408c1d700 1 -- 172.21.15.91:0/4043810615 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f00035b0 con 0x7fd4040cf2b0 2023-11-24T06:09:24.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd408c1d700 1 -- 172.21.15.91:0/4043810615 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1481829282 0 0) 0x7fd3f00035b0 con 0x7fd4040cf2b0 2023-11-24T06:09:24.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd408c1d700 1 -- 172.21.15.91:0/4043810615 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4040ce5c0 msgr2=0x7fd4040cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:24.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd408c1d700 1 --2- 172.21.15.91:0/4043810615 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4040ce5c0 0x7fd4040cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd408c1d700 1 -- 172.21.15.91:0/4043810615 >> v2:172.21.15.94:3300/0 conn(0x7fd40415bde0 msgr2=0x7fd40415c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:24.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd408c1d700 1 --2- 172.21.15.91:0/4043810615 >> v2:172.21.15.94:3300/0 conn(0x7fd40415bde0 0x7fd40415c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd408c1d700 1 -- 172.21.15.91:0/4043810615 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40406f0f0 con 0x7fd4040cf2b0 2023-11-24T06:09:24.137 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd408c1d700 1 -- 172.21.15.91:0/4043810615 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7fd400001670 con 0x7fd4040cf2b0 2023-11-24T06:09:24.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd408c1d700 1 -- 172.21.15.91:0/4043810615 >> v1:172.21.15.91:6789/0 conn(0x7fd4040cf2b0 legacy=0x7fd4040d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:24.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd408c1d700 1 --2- 172.21.15.91:0/4043810615 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4040ce5c0 0x7fd3f0003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:24.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd408c1d700 1 --2- 172.21.15.91:0/4043810615 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd3f0003f80 0x7fd3f0004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:24.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd408c1d700 1 --2- 172.21.15.91:0/4043810615 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f0004d90 0x7fd3f0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:24.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd408c1d700 1 -- 172.21.15.91:0/4043810615 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4040cefb0 con 0x7fd3f0003f80 2023-11-24T06:09:24.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd408c1d700 1 -- 172.21.15.91:0/4043810615 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40406ee10 con 0x7fd3f0004d90 2023-11-24T06:09:24.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd408c1d700 1 -- 172.21.15.91:0/4043810615 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f000ba60 con 0x7fd4040ce5c0 2023-11-24T06:09:24.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd40941e700 1 --2- 172.21.15.91:0/4043810615 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd3f0003f80 0x7fd3f0004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:24.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd40a420700 1 --2- 172.21.15.91:0/4043810615 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f0004d90 0x7fd3f0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:24.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd409c1f700 1 --2- 172.21.15.91:0/4043810615 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4040ce5c0 0x7fd3f0003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:24.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd40941e700 1 -- 172.21.15.91:0/4043810615 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4040ce5c0 msgr2=0x7fd3f0003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:24.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd40941e700 1 --2- 172.21.15.91:0/4043810615 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4040ce5c0 0x7fd3f0003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.138 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd40941e700 1 -- 172.21.15.91:0/4043810615 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f0004d90 msgr2=0x7fd3f0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:24.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd40941e700 1 --2- 172.21.15.91:0/4043810615 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f0004d90 0x7fd3f0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd40941e700 1 -- 172.21.15.91:0/4043810615 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f00035b0 con 0x7fd3f0003f80 2023-11-24T06:09:24.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd409c1f700 1 --2- 172.21.15.91:0/4043810615 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4040ce5c0 0x7fd3f0003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:24.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.130+0000 7fd40941e700 1 --2- 172.21.15.91:0/4043810615 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd3f0003f80 0x7fd3f0004850 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd40000c370 tx=0x7fd3f0005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf7f1ab765a1032 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:24.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.131+0000 7fd408c1d700 1 -- 172.21.15.91:0/4043810615 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd400007c60 con 0x7fd3f0003f80 2023-11-24T06:09:24.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.131+0000 7fd408c1d700 1 -- 172.21.15.91:0/4043810615 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40000ce70 con 0x7fd3f0003f80 2023-11-24T06:09:24.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.131+0000 7fd408c1d700 1 -- 172.21.15.91:0/4043810615 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd400005b80 con 0x7fd3f0003f80 2023-11-24T06:09:24.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.131+0000 7fd40be83700 1 -- 172.21.15.91:0/4043810615 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd3f0003f80 msgr2=0x7fd3f0004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:24.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.131+0000 7fd40be83700 1 --2- 172.21.15.91:0/4043810615 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd3f0003f80 0x7fd3f0004850 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd40000c370 tx=0x7fd3f0005880 comp rx=0 tx=0).stop 2023-11-24T06:09:24.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.131+0000 7fd40a420700 1 -- 172.21.15.91:0/4043810615 reap_dead start 2023-11-24T06:09:24.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.131+0000 7fd40be83700 1 -- 172.21.15.91:0/4043810615 shutdown_connections 2023-11-24T06:09:24.139 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.131+0000 7fd40be83700 1 -- 172.21.15.91:0/4043810615 >> 172.21.15.91:0/4043810615 conn(0x7fd4040b3720 msgr2=0x7fd4040b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:24.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.131+0000 7fd40be83700 1 -- 172.21.15.91:0/4043810615 shutdown_connections 2023-11-24T06:09:24.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.131+0000 7fd40be83700 1 -- 172.21.15.91:0/4043810615 wait complete. 2023-11-24T06:09:24.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.132+0000 7fd40be83700 1 Processor -- start 2023-11-24T06:09:24.140 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.132+0000 7fd40be83700 1 -- start start 2023-11-24T06:09:24.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.133+0000 7fd40be83700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f0003f80 0x7fd4040b5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:24.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.133+0000 7fd40be83700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4040b6c50 0x7fd4041f5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:24.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.133+0000 7fd40be83700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4041f26c0 0x7fd4040ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:24.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.133+0000 7fd40be83700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4041f19f0 con 0x7fd4040b6c50 2023-11-24T06:09:24.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.133+0000 7fd40be83700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4041f2230 con 0x7fd3f0003f80 2023-11-24T06:09:24.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.133+0000 7fd40be83700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4040b61f0 con 0x7fd4041f26c0 2023-11-24T06:09:24.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.133+0000 7fd40941e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4040b6c50 0x7fd4041f5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:24.142 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.133+0000 7fd40941e700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4040b6c50 0x7fd4041f5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60310/0 (socket says 172.21.15.91:60310) 2023-11-24T06:09:24.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.133+0000 7fd40941e700 1 -- 172.21.15.91:0/2906690442 learned_addr learned my addr 172.21.15.91:0/2906690442 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:24.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.133+0000 7fd40a420700 1 --2- 172.21.15.91:0/2906690442 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4041f26c0 0x7fd4040ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:24.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.133+0000 7fd409c1f700 1 --2- 172.21.15.91:0/2906690442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f0003f80 0x7fd4040b5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:24.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.133+0000 7fd40941e700 1 -- 172.21.15.91:0/2906690442 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4041f26c0 msgr2=0x7fd4040ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:24.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.133+0000 7fd40941e700 1 --2- 172.21.15.91:0/2906690442 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4041f26c0 0x7fd4040ca940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.133+0000 7fd40941e700 1 -- 172.21.15.91:0/2906690442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f0003f80 msgr2=0x7fd4040b5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:24.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.133+0000 7fd40941e700 1 --2- 172.21.15.91:0/2906690442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f0003f80 0x7fd4040b5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.133+0000 7fd40941e700 1 -- 172.21.15.91:0/2906690442 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4040b70e0 con 0x7fd4040b6c50 2023-11-24T06:09:24.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.134+0000 7fd40941e700 1 --2- 172.21.15.91:0/2906690442 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4040b6c50 0x7fd4041f5880 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd40000c960 tx=0x7fd400000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:24.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.134+0000 7fd3fa7fc700 1 -- 172.21.15.91:0/2906690442 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4000070d0 con 0x7fd4040b6c50 2023-11-24T06:09:24.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.134+0000 7fd3fa7fc700 1 -- 172.21.15.91:0/2906690442 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd400007230 con 0x7fd4040b6c50 2023-11-24T06:09:24.143 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.134+0000 7fd40be83700 1 -- 172.21.15.91:0/2906690442 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4041f1be0 con 0x7fd4040b6c50 2023-11-24T06:09:24.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.134+0000 7fd3fa7fc700 1 -- 172.21.15.91:0/2906690442 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4000075e0 con 0x7fd4040b6c50 2023-11-24T06:09:24.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.134+0000 7fd40be83700 1 -- 172.21.15.91:0/2906690442 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4041f5dc0 con 0x7fd4040b6c50 2023-11-24T06:09:24.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.135+0000 7fd40be83700 1 -- 172.21.15.91:0/2906690442 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd404075480 con 0x7fd4040b6c50 2023-11-24T06:09:24.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.135+0000 7fd3fa7fc700 1 -- 172.21.15.91:0/2906690442 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fd40001e430 con 0x7fd4040b6c50 2023-11-24T06:09:24.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.136+0000 7fd3fa7fc700 1 --2- 172.21.15.91:0/2906690442 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd3e806af60 0x7fd3e806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:24.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.136+0000 7fd3fa7fc700 1 -- 172.21.15.91:0/2906690442 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd40008aa90 con 0x7fd4040b6c50 2023-11-24T06:09:24.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.136+0000 7fd409c1f700 1 --2- 172.21.15.91:0/2906690442 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd3e806af60 0x7fd3e806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:24.144 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.136+0000 7fd409c1f700 1 --2- 172.21.15.91:0/2906690442 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd3e806af60 0x7fd3e806d420 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4009a30 tx=0x7fd3f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:24.145 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.140+0000 7fd3fa7fc700 1 -- 172.21.15.91:0/2906690442 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd400056000 con 0x7fd4040b6c50 2023-11-24T06:09:24.342 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.339+0000 7fd40be83700 1 -- 172.21.15.91:0/2906690442 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd4040b73c0 con 0x7fd4040b6c50 2023-11-24T06:09:24.345 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.342+0000 7fd3fa7fc700 1 -- 172.21.15.91:0/2906690442 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd400007dc0 con 0x7fd4040b6c50 2023-11-24T06:09:24.348 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.344+0000 7fd40be83700 1 -- 172.21.15.91:0/2906690442 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd3e806af60 msgr2=0x7fd3e806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:24.348 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.344+0000 7fd40be83700 1 --2- 172.21.15.91:0/2906690442 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd3e806af60 0x7fd3e806d420 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4009a30 tx=0x7fd3f4008040 comp rx=0 tx=0).stop 2023-11-24T06:09:24.348 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.344+0000 7fd40be83700 1 -- 172.21.15.91:0/2906690442 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4040b6c50 msgr2=0x7fd4041f5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:24.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.344+0000 7fd40be83700 1 --2- 172.21.15.91:0/2906690442 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4040b6c50 0x7fd4041f5880 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd40000c960 tx=0x7fd400000fe0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.344+0000 7fd40be83700 1 -- 172.21.15.91:0/2906690442 shutdown_connections 2023-11-24T06:09:24.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.344+0000 7fd40be83700 1 --2- 172.21.15.91:0/2906690442 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fd3e806af60 0x7fd3e806d420 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.344+0000 7fd40be83700 1 --2- 172.21.15.91:0/2906690442 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fd3f0003f80 0x7fd4040b5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.344+0000 7fd40be83700 1 --2- 172.21.15.91:0/2906690442 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fd4041f26c0 0x7fd4040ca940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.344+0000 7fd40be83700 1 --2- 172.21.15.91:0/2906690442 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fd4040b6c50 0x7fd4041f5880 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.344+0000 7fd40be83700 1 -- 172.21.15.91:0/2906690442 >> 172.21.15.91:0/2906690442 conn(0x7fd4040b3720 msgr2=0x7fd4040ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:24.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.344+0000 7fd40be83700 1 -- 172.21.15.91:0/2906690442 shutdown_connections 2023-11-24T06:09:24.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.344+0000 7fd40be83700 1 -- 172.21.15.91:0/2906690442 wait complete. 2023-11-24T06:09:24.360 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_failed' 2023-11-24T06:09:24.481 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.477+0000 7f47ad2ba700 1 Processor -- start 2023-11-24T06:09:24.481 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.477+0000 7f47ad2ba700 1 -- start start 2023-11-24T06:09:24.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.478+0000 7f47ad2ba700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47a815e460 0x7f47a80d49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:24.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.478+0000 7f47ad2ba700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f47a80cf480 0x7f47a80d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:24.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.478+0000 7f47ad2ba700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a80b3370 con 0x7f47a80d29d0 2023-11-24T06:09:24.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.478+0000 7f47ad2ba700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a80d2e60 con 0x7f47a80cf480 2023-11-24T06:09:24.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.478+0000 7f47ad2ba700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a80cfc40 con 0x7f47a815e460 2023-11-24T06:09:24.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.478+0000 7f47a67fc700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f47a80cf480 0x7f47a80d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:24.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.478+0000 7f47a67fc700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f47a80cf480 0x7f47a80d2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60316/0 (socket says 172.21.15.91:60316) 2023-11-24T06:09:24.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.478+0000 7f47a67fc700 1 -- 172.21.15.91:0/1283177821 learned_addr learned my addr 172.21.15.91:0/1283177821 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:24.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.478+0000 7f47a6ffd700 1 --2- 172.21.15.91:0/1283177821 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47a815e460 0x7f47a80d49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:24.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.478+0000 7f47a67fc700 1 -- 172.21.15.91:0/1283177821 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47a815e460 msgr2=0x7f47a80d49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:24.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.478+0000 7f47a67fc700 1 --2- 172.21.15.91:0/1283177821 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47a815e460 0x7f47a80d49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.478+0000 7f47a67fc700 1 -- 172.21.15.91:0/1283177821 >> v1:172.21.15.94:6789/0 conn(0x7f47a80d29d0 legacy=0x7f47a80d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:24.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.478+0000 7f47a67fc700 1 -- 172.21.15.91:0/1283177821 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a80b35c0 con 0x7f47a80cf480 2023-11-24T06:09:24.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.479+0000 7f47a67fc700 1 --2- 172.21.15.91:0/1283177821 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f47a80cf480 0x7f47a80d2490 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f47980098f0 tx=0x7f4798009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bdf6766e07df704 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:24.484 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.479+0000 7f47a5ffb700 1 -- 172.21.15.91:0/1283177821 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4798004230 con 0x7f47a80cf480 2023-11-24T06:09:24.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.479+0000 7f47a5ffb700 1 -- 172.21.15.91:0/1283177821 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4798004390 con 0x7f47a80cf480 2023-11-24T06:09:24.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.479+0000 7f47a5ffb700 1 -- 172.21.15.91:0/1283177821 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47980036a0 con 0x7f47a80cf480 2023-11-24T06:09:24.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.479+0000 7f47ad2ba700 1 -- 172.21.15.91:0/1283177821 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f47a80cf480 msgr2=0x7f47a80d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:24.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.479+0000 7f47ad2ba700 1 --2- 172.21.15.91:0/1283177821 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f47a80cf480 0x7f47a80d2490 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f47980098f0 tx=0x7f4798009c00 comp rx=0 tx=0).stop 2023-11-24T06:09:24.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.479+0000 7f47ad2ba700 1 -- 172.21.15.91:0/1283177821 shutdown_connections 2023-11-24T06:09:24.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.479+0000 7f47ad2ba700 1 --2- 172.21.15.91:0/1283177821 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47a815e460 0x7f47a80d49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.479+0000 7f47ad2ba700 1 --2- 172.21.15.91:0/1283177821 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f47a80cf480 0x7f47a80d2490 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.479+0000 7f47ad2ba700 1 -- 172.21.15.91:0/1283177821 >> 172.21.15.91:0/1283177821 conn(0x7f47a80d3eb0 msgr2=0x7f47a80d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:24.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.479+0000 7f47ad2ba700 1 -- 172.21.15.91:0/1283177821 shutdown_connections 2023-11-24T06:09:24.485 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.479+0000 7f47ad2ba700 1 -- 172.21.15.91:0/1283177821 wait complete. 2023-11-24T06:09:24.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.481+0000 7f47ad2ba700 1 Processor -- start 2023-11-24T06:09:24.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.481+0000 7f47ad2ba700 1 -- start start 2023-11-24T06:09:24.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.481+0000 7f47ad2ba700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47a80cf480 0x7f47a80cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:24.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.481+0000 7f47ad2ba700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47a80d29d0 0x7f47a8161360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:24.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.481+0000 7f47ad2ba700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f47a815e460 0x7f47a81f3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:24.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.481+0000 7f47ad2ba700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a81604a0 con 0x7f47a815e460 2023-11-24T06:09:24.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.481+0000 7f47ad2ba700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a80b83e0 con 0x7f47a80d29d0 2023-11-24T06:09:24.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.481+0000 7f47ad2ba700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a80cd000 con 0x7f47a80cf480 2023-11-24T06:09:24.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f47a77fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f47a815e460 0x7f47a81f3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:24.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f47a77fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f47a815e460 0x7f47a81f3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60324/0 (socket says 172.21.15.91:60324) 2023-11-24T06:09:24.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f47a77fe700 1 -- 172.21.15.91:0/3449145552 learned_addr learned my addr 172.21.15.91:0/3449145552 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:24.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f47a67fc700 1 --2- 172.21.15.91:0/3449145552 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47a80d29d0 0x7f47a8161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:24.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f47a6ffd700 1 --2- 172.21.15.91:0/3449145552 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47a80cf480 0x7f47a80cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:24.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f47a77fe700 1 -- 172.21.15.91:0/3449145552 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47a80cf480 msgr2=0x7f47a80cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:24.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f47a77fe700 1 --2- 172.21.15.91:0/3449145552 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47a80cf480 0x7f47a80cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f47a77fe700 1 -- 172.21.15.91:0/3449145552 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47a80d29d0 msgr2=0x7f47a8161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:24.490 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f47a77fe700 1 --2- 172.21.15.91:0/3449145552 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47a80d29d0 0x7f47a8161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.490 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f47a77fe700 1 -- 172.21.15.91:0/3449145552 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4798009550 con 0x7f47a815e460 2023-11-24T06:09:24.490 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f47a67fc700 1 --2- 172.21.15.91:0/3449145552 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47a80d29d0 0x7f47a8161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:24.490 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f47a6ffd700 1 --2- 172.21.15.91:0/3449145552 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47a80cf480 0x7f47a80cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:24.490 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f47a77fe700 1 --2- 172.21.15.91:0/3449145552 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f47a815e460 0x7f47a81f3cb0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f47a000cc10 tx=0x7f47a000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:24.490 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f478f7fe700 1 -- 172.21.15.91:0/3449145552 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a000abb0 con 0x7f47a815e460 2023-11-24T06:09:24.490 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f478f7fe700 1 -- 172.21.15.91:0/3449145552 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a000ad10 con 0x7f47a815e460 2023-11-24T06:09:24.490 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f47ad2ba700 1 -- 172.21.15.91:0/3449145552 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a80cd2e0 con 0x7f47a815e460 2023-11-24T06:09:24.490 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f478f7fe700 1 -- 172.21.15.91:0/3449145552 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a0012660 con 0x7f47a815e460 2023-11-24T06:09:24.490 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.482+0000 7f47ad2ba700 1 -- 172.21.15.91:0/3449145552 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a80cd8a0 con 0x7f47a815e460 2023-11-24T06:09:24.490 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.483+0000 7f47ad2ba700 1 -- 172.21.15.91:0/3449145552 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a80b81b0 con 0x7f47a815e460 2023-11-24T06:09:24.490 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.484+0000 7f478f7fe700 1 -- 172.21.15.91:0/3449145552 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f47a00127c0 con 0x7f47a815e460 2023-11-24T06:09:24.490 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.484+0000 7f478f7fe700 1 --2- 172.21.15.91:0/3449145552 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f479406af60 0x7f479406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:24.491 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.484+0000 7f478f7fe700 1 -- 172.21.15.91:0/3449145552 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f47a0089a70 con 0x7f47a815e460 2023-11-24T06:09:24.491 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.484+0000 7f47a6ffd700 1 --2- 172.21.15.91:0/3449145552 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f479406af60 0x7f479406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:24.491 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.484+0000 7f47a6ffd700 1 --2- 172.21.15.91:0/3449145552 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f479406af60 0x7f479406d420 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f479c007a50 tx=0x7f479c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:24.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.488+0000 7f478f7fe700 1 -- 172.21.15.91:0/3449145552 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47a80b81b0 con 0x7f47a815e460 2023-11-24T06:09:24.761 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.757+0000 7f47ad2ba700 1 -- 172.21.15.91:0/3449145552 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_failed"]} v 0) v1 -- 0x7f47a80b81b0 con 0x7f47a815e460 2023-11-24T06:09:24.991 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.987+0000 7f478f7fe700 1 -- 172.21.15.91:0/3449145552 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones.test_clone_failure_status_failed"]}]=0 v103) v1 ==== 165+0+0 (secure 0 0 0) 0x7f47a80b81b0 con 0x7f47a815e460 2023-11-24T06:09:24.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.990+0000 7f47ad2ba700 1 -- 172.21.15.91:0/3449145552 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f479406af60 msgr2=0x7f479406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:24.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.990+0000 7f47ad2ba700 1 --2- 172.21.15.91:0/3449145552 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f479406af60 0x7f479406d420 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f479c007a50 tx=0x7f479c008040 comp rx=0 tx=0).stop 2023-11-24T06:09:24.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.990+0000 7f47ad2ba700 1 -- 172.21.15.91:0/3449145552 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f47a815e460 msgr2=0x7f47a81f3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:24.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.990+0000 7f47ad2ba700 1 --2- 172.21.15.91:0/3449145552 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f47a815e460 0x7f47a81f3cb0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f47a000cc10 tx=0x7f47a000a060 comp rx=0 tx=0).stop 2023-11-24T06:09:24.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.990+0000 7f47ad2ba700 1 -- 172.21.15.91:0/3449145552 shutdown_connections 2023-11-24T06:09:24.994 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.990+0000 7f47ad2ba700 1 --2- 172.21.15.91:0/3449145552 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f479406af60 0x7f479406d420 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.990+0000 7f47ad2ba700 1 --2- 172.21.15.91:0/3449145552 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f47a815e460 0x7f47a81f3cb0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.990+0000 7f47ad2ba700 1 --2- 172.21.15.91:0/3449145552 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f47a80cf480 0x7f47a80cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.990+0000 7f47ad2ba700 1 --2- 172.21.15.91:0/3449145552 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f47a80d29d0 0x7f47a8161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:24.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.990+0000 7f47ad2ba700 1 -- 172.21.15.91:0/3449145552 >> 172.21.15.91:0/3449145552 conn(0x7f47a80d3eb0 msgr2=0x7f47a80b1340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:24.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.990+0000 7f47ad2ba700 1 -- 172.21.15.91:0/3449145552 shutdown_connections 2023-11-24T06:09:24.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:24.990+0000 7f47ad2ba700 1 -- 172.21.15.91:0/3449145552 wait complete. 2023-11-24T06:09:25.008 INFO:tasks.cephfs_test_runner:ensure failure status is shown when clone is in failed state and validate the reason ... ERROR 2023-11-24T06:09:25.008 INFO:tasks.cephfs_test_runner: 2023-11-24T06:09:25.008 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-24T06:09:25.009 INFO:tasks.cephfs_test_runner:ERROR: test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T06:09:25.009 INFO:tasks.cephfs_test_runner:ensure failure status is shown when clone is in failed state and validate the reason 2023-11-24T06:09:25.009 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-24T06:09:25.009 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-24T06:09:25.009 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa/tasks/cephfs/test_volumes.py", line 6608, in test_clone_failure_status_failed 2023-11-24T06:09:25.009 INFO:tasks.cephfs_test_runner: self._do_subvolume_io(subvolume, number_of_files=200) 2023-11-24T06:09:25.009 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa/tasks/cephfs/test_volumes.py", line 278, in _do_subvolume_io 2023-11-24T06:09:25.009 INFO:tasks.cephfs_test_runner: filename = "{0}.{1}".format(TestVolumes.TEST_FILE_NAME_PREFIX, i) 2023-11-24T06:09:25.009 INFO:tasks.cephfs_test_runner:AttributeError: type object 'TestVolumes' has no attribute 'TEST_FILE_NAME_PREFIX' 2023-11-24T06:09:25.009 INFO:tasks.cephfs_test_runner: 2023-11-24T06:09:25.009 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-24T06:09:25.009 INFO:tasks.cephfs_test_runner:Ran 1 test in 52.264s 2023-11-24T06:09:25.009 INFO:tasks.cephfs_test_runner: 2023-11-24T06:09:25.010 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2023-11-24T06:09:25.010 INFO:tasks.cephfs_test_runner: 2023-11-24T06:09:25.010 INFO:tasks.cephfs_test_runner:====================================================================== 2023-11-24T06:09:25.010 INFO:tasks.cephfs_test_runner:ERROR: test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T06:09:25.010 INFO:tasks.cephfs_test_runner:ensure failure status is shown when clone is in failed state and validate the reason 2023-11-24T06:09:25.010 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-11-24T06:09:25.010 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-11-24T06:09:25.010 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa/tasks/cephfs/test_volumes.py", line 6608, in test_clone_failure_status_failed 2023-11-24T06:09:25.010 INFO:tasks.cephfs_test_runner: self._do_subvolume_io(subvolume, number_of_files=200) 2023-11-24T06:09:25.010 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa/tasks/cephfs/test_volumes.py", line 278, in _do_subvolume_io 2023-11-24T06:09:25.010 INFO:tasks.cephfs_test_runner: filename = "{0}.{1}".format(TestVolumes.TEST_FILE_NAME_PREFIX, i) 2023-11-24T06:09:25.010 INFO:tasks.cephfs_test_runner:AttributeError: type object 'TestVolumes' has no attribute 'TEST_FILE_NAME_PREFIX' 2023-11-24T06:09:25.011 INFO:tasks.cephfs_test_runner: 2023-11-24T06:09:25.011 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T06:09:25.075 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e29d0b327c4647df8c3c77dcdd0507f6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_a13ea5ff00cb4df7a9cc941166ddeb358881144c/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2023-11-24T06:09:25.077 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-11-24T06:09:25.085 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-11-24T06:09:25.103 INFO:tasks.kclient:Unmounting kernel clients... 2023-11-24T06:09:25.103 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:09:25.103 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:09:25.134 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13970 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-24T06:09:25.134 DEBUG:teuthology.orchestra.run.smithi091:> set -ex 2023-11-24T06:09:25.134 DEBUG:teuthology.orchestra.run.smithi091:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:09:25.191 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284204k,nr_inodes=4071051,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13970 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260540k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw,seclabel 0 0 2023-11-24T06:09:25.191 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:09:25.191 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:09:25.212 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16729 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-24T06:09:25.212 DEBUG:teuthology.orchestra.run.smithi094:> set -ex 2023-11-24T06:09:25.213 DEBUG:teuthology.orchestra.run.smithi094:> dd if=/proc/self/mounts of=/dev/stdout 2023-11-24T06:09:25.273 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16284224k,nr_inodes=4071056,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,seclabel,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16729 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3260544k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw,seclabel 0 0 2023-11-24T06:09:25.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:25.273 DEBUG:teuthology.orchestra.run.smithi091:> ip netns list 2023-11-24T06:09:25.290 INFO:teuthology.orchestra.run.smithi091.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-11-24T06:09:25.291 INFO:teuthology.orchestra.run.smithi091.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-11-24T06:09:25.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:25.291 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-11-24T06:09:25.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:25.379 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-11-24T06:09:25.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:25.462 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip link delete ceph-brx 2023-11-24T06:09:25.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:25.552 DEBUG:teuthology.orchestra.run.smithi091:> ip netns list 2023-11-24T06:09:25.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:25.570 DEBUG:teuthology.orchestra.run.smithi091:> sudo ip link delete ceph-brx 2023-11-24T06:09:25.639 INFO:teuthology.orchestra.run.smithi091.stderr:Cannot find device "ceph-brx" 2023-11-24T06:09:25.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:09:25.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:25.641 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2023-11-24T06:09:25.661 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-11-24T06:09:25.661 INFO:teuthology.orchestra.run.smithi094.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-11-24T06:09:25.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:25.662 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-11-24T06:09:25.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:25.743 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-11-24T06:09:25.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:25.835 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link delete ceph-brx 2023-11-24T06:09:25.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:25.928 DEBUG:teuthology.orchestra.run.smithi094:> ip netns list 2023-11-24T06:09:25.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-11-24T06:09:25.944 DEBUG:teuthology.orchestra.run.smithi094:> sudo ip link delete ceph-brx 2023-11-24T06:09:26.014 INFO:teuthology.orchestra.run.smithi094.stderr:Cannot find device "ceph-brx" 2023-11-24T06:09:26.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:09:26.015 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-24T06:09:26.024 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-24T06:09:26.025 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:09:26.146 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.143+0000 7f2bd3586700 1 Processor -- start 2023-11-24T06:09:26.147 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.143+0000 7f2bd3586700 1 -- start start 2023-11-24T06:09:26.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.144+0000 7f2bd3586700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f2bcc0cf1e0 0x7f2bcc15b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.144+0000 7f2bd3586700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bcc072d20 con 0x7f2bcc070a80 2023-11-24T06:09:26.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.144+0000 7f2bd3586700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bcc071020 con 0x7f2bcc0ca6e0 2023-11-24T06:09:26.149 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.144+0000 7f2bd3586700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bcc15ba10 con 0x7f2bcc0cf1e0 2023-11-24T06:09:26.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.144+0000 7f2bd1322700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f2bcc0cf1e0 0x7f2bcc15b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.144+0000 7f2bd0b21700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f2bcc070a80 0x7f2bcc0c9fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55266/0 (socket says 172.21.15.91:55266) 2023-11-24T06:09:26.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.144+0000 7f2bd1322700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f2bcc0cf1e0 0x7f2bcc15b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60330/0 (socket says 172.21.15.91:60330) 2023-11-24T06:09:26.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.144+0000 7f2bd0b21700 1 -- 172.21.15.91:0/455102139 learned_addr learned my addr 172.21.15.91:0/455102139 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:26.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.144+0000 7f2bd1322700 1 -- 172.21.15.91:0/455102139 >> v1:172.21.15.94:6789/0 conn(0x7f2bcc0ca6e0 legacy=0x7f2bcc06d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:26.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.144+0000 7f2bd1322700 1 -- 172.21.15.91:0/455102139 >> v1:172.21.15.91:6789/0 conn(0x7f2bcc070a80 legacy=0x7f2bcc0c9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:26.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.144+0000 7f2bd1322700 1 -- 172.21.15.91:0/455102139 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc071300 con 0x7f2bcc0cf1e0 2023-11-24T06:09:26.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.144+0000 7f2bc3fff700 1 -- 172.21.15.91:0/455102139 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184589853 0 0) 0x7f2bcc072d20 con 0x7f2bcc070a80 2023-11-24T06:09:26.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.144+0000 7f2bd1322700 1 --2- 172.21.15.91:0/455102139 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2bcc0cf1e0 0x7f2bcc15b4d0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009840 tx=0x7f2bbc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2886962a95f466a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:26.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.144+0000 7f2bc3fff700 1 -- 172.21.15.91:0/455102139 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc011830 con 0x7f2bcc0cf1e0 2023-11-24T06:09:26.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.144+0000 7f2bc3fff700 1 -- 172.21.15.91:0/455102139 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bbc011990 con 0x7f2bcc0cf1e0 2023-11-24T06:09:26.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.144+0000 7f2bc3fff700 1 -- 172.21.15.91:0/455102139 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc011d20 con 0x7f2bcc0cf1e0 2023-11-24T06:09:26.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.145+0000 7f2bd3586700 1 -- 172.21.15.91:0/455102139 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2bcc0cf1e0 msgr2=0x7f2bcc15b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:26.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.145+0000 7f2bd3586700 1 --2- 172.21.15.91:0/455102139 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2bcc0cf1e0 0x7f2bcc15b4d0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009840 tx=0x7f2bbc009b50 comp rx=0 tx=0).stop 2023-11-24T06:09:26.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.145+0000 7f2bd3586700 1 -- 172.21.15.91:0/455102139 shutdown_connections 2023-11-24T06:09:26.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.145+0000 7f2bd3586700 1 --2- 172.21.15.91:0/455102139 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2bcc0cf1e0 0x7f2bcc15b4d0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.152 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.145+0000 7f2bd3586700 1 -- 172.21.15.91:0/455102139 >> 172.21.15.91:0/455102139 conn(0x7f2bcc0cf980 msgr2=0x7f2bcc0cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:26.152 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.145+0000 7f2bd3586700 1 -- 172.21.15.91:0/455102139 shutdown_connections 2023-11-24T06:09:26.152 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.145+0000 7f2bd3586700 1 -- 172.21.15.91:0/455102139 wait complete. 2023-11-24T06:09:26.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.146+0000 7f2bd3586700 1 Processor -- start 2023-11-24T06:09:26.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.146+0000 7f2bd3586700 1 -- start start 2023-11-24T06:09:26.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.147+0000 7f2bd3586700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2bcc070a80 0x7f2bcc147350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.147+0000 7f2bd3586700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2bcc0ca6e0 0x7f2bcc142c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.147+0000 7f2bd3586700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bcc0cf1e0 0x7f2bcc150690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.147+0000 7f2bd3586700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc0b3180 con 0x7f2bcc070a80 2023-11-24T06:09:26.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.147+0000 7f2bd3586700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc0b2590 con 0x7f2bcc0cf1e0 2023-11-24T06:09:26.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.147+0000 7f2bd3586700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc0b27e0 con 0x7f2bcc0ca6e0 2023-11-24T06:09:26.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.147+0000 7f2bd1322700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2bcc070a80 0x7f2bcc147350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.147+0000 7f2bd1322700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2bcc070a80 0x7f2bcc147350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60336/0 (socket says 172.21.15.91:60336) 2023-11-24T06:09:26.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.147+0000 7f2bd1322700 1 -- 172.21.15.91:0/1320999817 learned_addr learned my addr 172.21.15.91:0/1320999817 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:26.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.147+0000 7f2bd0b21700 1 --2- 172.21.15.91:0/1320999817 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2bcc0ca6e0 0x7f2bcc142c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.147+0000 7f2bd1b23700 1 --2- 172.21.15.91:0/1320999817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bcc0cf1e0 0x7f2bcc150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.147+0000 7f2bd1322700 1 -- 172.21.15.91:0/1320999817 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2bcc0ca6e0 msgr2=0x7f2bcc142c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:26.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.147+0000 7f2bd1322700 1 --2- 172.21.15.91:0/1320999817 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2bcc0ca6e0 0x7f2bcc142c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.147+0000 7f2bd1322700 1 -- 172.21.15.91:0/1320999817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bcc0cf1e0 msgr2=0x7f2bcc150690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:26.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.147+0000 7f2bd1322700 1 --2- 172.21.15.91:0/1320999817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bcc0cf1e0 0x7f2bcc150690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.147+0000 7f2bd1322700 1 -- 172.21.15.91:0/1320999817 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc0094a0 con 0x7f2bcc070a80 2023-11-24T06:09:26.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.148+0000 7f2bd1322700 1 --2- 172.21.15.91:0/1320999817 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2bcc070a80 0x7f2bcc147350 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc014040 tx=0x7f2bbc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:26.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.148+0000 7f2bc1ffb700 1 -- 172.21.15.91:0/1320999817 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc01a860 con 0x7f2bcc070a80 2023-11-24T06:09:26.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.148+0000 7f2bc1ffb700 1 -- 172.21.15.91:0/1320999817 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bbc01a9e0 con 0x7f2bcc070a80 2023-11-24T06:09:26.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.148+0000 7f2bd3586700 1 -- 172.21.15.91:0/1320999817 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bcc0b2ac0 con 0x7f2bcc070a80 2023-11-24T06:09:26.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.148+0000 7f2bc1ffb700 1 -- 172.21.15.91:0/1320999817 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc01ad90 con 0x7f2bcc070a80 2023-11-24T06:09:26.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.148+0000 7f2bd3586700 1 -- 172.21.15.91:0/1320999817 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bcc1431c0 con 0x7f2bcc070a80 2023-11-24T06:09:26.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.149+0000 7f2bd3586700 1 -- 172.21.15.91:0/1320999817 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bcc06cac0 con 0x7f2bcc070a80 2023-11-24T06:09:26.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.149+0000 7f2bc1ffb700 1 -- 172.21.15.91:0/1320999817 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f2bbc022070 con 0x7f2bcc070a80 2023-11-24T06:09:26.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.150+0000 7f2bc1ffb700 1 --2- 172.21.15.91:0/1320999817 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2bb006af60 0x7f2bb006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.150+0000 7f2bc1ffb700 1 -- 172.21.15.91:0/1320999817 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2bbc08a4e0 con 0x7f2bcc070a80 2023-11-24T06:09:26.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.150+0000 7f2bd0b21700 1 --2- 172.21.15.91:0/1320999817 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2bb006af60 0x7f2bb006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.150+0000 7f2bd0b21700 1 --2- 172.21.15.91:0/1320999817 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2bb006af60 0x7f2bb006d420 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8001340 tx=0x7f2bc80012d0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:26.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.155+0000 7f2bc1ffb700 1 -- 172.21.15.91:0/1320999817 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2bcc06cac0 con 0x7f2bcc070a80 2023-11-24T06:09:26.343 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.340+0000 7f2bd3586700 1 -- 172.21.15.91:0/1320999817 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2bcc0b2950 con 0x7f2bb006af60 2023-11-24T06:09:26.345 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.342+0000 7f2bc1ffb700 1 -- 172.21.15.91:0/1320999817 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50191 (secure 0 0 0) 0x7f2bcc0b2950 con 0x7f2bb006af60 2023-11-24T06:09:26.346 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:26.348 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.344+0000 7f2bd3586700 1 -- 172.21.15.91:0/1320999817 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2bb006af60 msgr2=0x7f2bb006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:26.348 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.344+0000 7f2bd3586700 1 --2- 172.21.15.91:0/1320999817 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2bb006af60 0x7f2bb006d420 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8001340 tx=0x7f2bc80012d0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.348 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.344+0000 7f2bd3586700 1 -- 172.21.15.91:0/1320999817 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2bcc070a80 msgr2=0x7f2bcc147350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:26.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.344+0000 7f2bd3586700 1 --2- 172.21.15.91:0/1320999817 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2bcc070a80 0x7f2bcc147350 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc014040 tx=0x7f2bbc004750 comp rx=0 tx=0).stop 2023-11-24T06:09:26.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.344+0000 7f2bd3586700 1 -- 172.21.15.91:0/1320999817 shutdown_connections 2023-11-24T06:09:26.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.344+0000 7f2bd3586700 1 --2- 172.21.15.91:0/1320999817 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f2bb006af60 0x7f2bb006d420 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.344+0000 7f2bd3586700 1 --2- 172.21.15.91:0/1320999817 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f2bcc0cf1e0 0x7f2bcc150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.344+0000 7f2bd3586700 1 --2- 172.21.15.91:0/1320999817 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f2bcc070a80 0x7f2bcc147350 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.344+0000 7f2bd3586700 1 --2- 172.21.15.91:0/1320999817 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f2bcc0ca6e0 0x7f2bcc142c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.349 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.344+0000 7f2bd3586700 1 -- 172.21.15.91:0/1320999817 >> 172.21.15.91:0/1320999817 conn(0x7f2bcc0cf980 msgr2=0x7f2bcc15bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:26.350 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.344+0000 7f2bd3586700 1 -- 172.21.15.91:0/1320999817 shutdown_connections 2023-11-24T06:09:26.350 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.344+0000 7f2bd3586700 1 -- 172.21.15.91:0/1320999817 wait complete. 2023-11-24T06:09:26.350 INFO:teuthology.orchestra.run.smithi091.stderr:dumped all 2023-11-24T06:09:26.360 INFO:teuthology.orchestra.run.smithi091.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2023-11-24T06:09:25.414072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220520,"kb_used_data":4352,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754200,"statfs":{"total":773094113280,"available":772868300800,"internally_reserved":0,"allocated":4456448,"data_stored":2496614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38149,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-68,"num_write_kb":-62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001185"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047270+0000","last_change":"2023-11-24T06:07:49.463719+0000","last_active":"2023-11-24T06:09:00.047270+0000","last_peered":"2023-11-24T06:09:00.047270+0000","last_clean":"2023-11-24T06:09:00.047270+0000","last_became_active":"2023-11-24T06:07:49.462736+0000","last_became_peered":"2023-11-24T06:07:49.462736+0000","last_unstale":"2023-11-24T06:09:00.047270+0000","last_undegraded":"2023-11-24T06:09:00.047270+0000","last_fullsized":"2023-11-24T06:09:00.047270+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_clean_scrub_stamp":"2023-11-24T06:07:47.860604+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:16.508770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":983040,"data_stored":742365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.432}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.499}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.45600000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.58999999999999997}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.55000000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.69899999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.72799999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":158604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.498},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.46300000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.51500000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.41099999999999998}]},{"osd":3,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.47999999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.318},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.36099999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.52800000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.443}]}]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":438272,"data_stored":189280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.7670000000000003,"5min":4.7670000000000003,"15min":4.7670000000000003},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.0810000000000004,"5min":6.0810000000000004,"15min":6.0810000000000004},"last":0.315}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.8340000000000001,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.375}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.35499999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.9529999999999998,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.60499999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.9640000000000004,"5min":5.9640000000000004,"15min":5.9640000000000004},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":3.02,"5min":3.02,"15min":3.02},"last":0.50900000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":6.048,"5min":6.048,"15min":6.048},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.9929999999999999,"5min":2.9929999999999999,"15min":2.9929999999999999},"last":0.57799999999999996}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":6.0129999999999999,"5min":6.0129999999999999,"15min":6.0129999999999999},"last":0.52100000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":748485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":12.076000000000001,"5min":12.076000000000001,"15min":12.076000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":4.181,"5min":4.181,"15min":4.181},"last":0.39600000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.2089999999999996,"5min":4.2089999999999996,"15min":4.2089999999999996},"last":0.435}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.30399999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":4.1520000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.59799999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":4.1260000000000003,"5min":4.1260000000000003,"15min":4.1260000000000003},"last":0.52600000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.73299999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.626}]}]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":158058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.7379999999999995,"5min":9.7379999999999995,"15min":9.7379999999999995},"last":0.309}]},{"osd":1,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":11.967000000000001,"5min":11.967000000000001,"15min":11.967000000000001},"last":0.252},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":9.593,"5min":9.593,"15min":9.593},"last":0.34399999999999997}]},{"osd":3,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.373},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.32600000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":9.6479999999999997,"5min":9.6479999999999997,"15min":9.6479999999999997},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.433}]},{"osd":5,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":9.7119999999999997,"5min":9.7119999999999997,"15min":9.7119999999999997},"last":0.502}]},{"osd":6,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.52000000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":9.6859999999999999,"5min":9.6859999999999999,"15min":9.6859999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.41599999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":434176,"data_stored":189151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.41199999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.38200000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.47899999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.36099999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.439},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.63700000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.505}]},{"osd":7,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.40300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":393216,"data_stored":152011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.9539999999999997,"5min":6.9539999999999997,"15min":6.9539999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":7.0039999999999996,"5min":7.0039999999999996,"15min":7.0039999999999996},"last":0.42699999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":12.907,"5min":12.907,"15min":12.907},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":6.9710000000000001,"5min":6.9710000000000001,"15min":6.9710000000000001},"last":0.371}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.51500000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":7.0540000000000003,"5min":7.0540000000000003,"15min":7.0540000000000003},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":7.0679999999999996,"5min":7.0679999999999996,"15min":7.0679999999999996},"last":0.499}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.52700000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55500000000000005}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.54300000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":158660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.31},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.38700000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.996,"5min":1.996,"15min":1.996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.55300000000000005}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":1.9079999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.47999999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.53200000000000003}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.41499999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.497},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.70799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:09:26.363 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:09:26.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.479+0000 7fba49707700 1 Processor -- start 2023-11-24T06:09:26.483 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.479+0000 7fba49707700 1 -- start start 2023-11-24T06:09:26.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.480+0000 7fba49707700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fba44143ae0 0x7fba440b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.480+0000 7fba49707700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fba441440f0 0x7fba44071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.480+0000 7fba49707700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba44143ec0 con 0x7fba440720f0 2023-11-24T06:09:26.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.480+0000 7fba49707700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba440b40a0 con 0x7fba441440f0 2023-11-24T06:09:26.486 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.480+0000 7fba49707700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba44072560 con 0x7fba44143ae0 2023-11-24T06:09:26.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.480+0000 7fba427fc700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fba441440f0 0x7fba44071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.480+0000 7fba437fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fba440720f0 0x7fba4415e8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55278/0 (socket says 172.21.15.91:55278) 2023-11-24T06:09:26.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.480+0000 7fba437fe700 1 -- 172.21.15.91:0/3299297733 learned_addr learned my addr 172.21.15.91:0/3299297733 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:26.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.481+0000 7fba42ffd700 1 --2- 172.21.15.91:0/3299297733 >> v2:172.21.15.94:3300/0 conn(0x7fba44143ae0 0x7fba440b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.481+0000 7fba427fc700 1 -- 172.21.15.91:0/3299297733 >> v2:172.21.15.94:3300/0 conn(0x7fba44143ae0 msgr2=0x7fba440b51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:26.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.481+0000 7fba427fc700 1 --2- 172.21.15.91:0/3299297733 >> v2:172.21.15.94:3300/0 conn(0x7fba44143ae0 0x7fba440b51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.481+0000 7fba427fc700 1 -- 172.21.15.91:0/3299297733 >> v1:172.21.15.91:6789/0 conn(0x7fba440720f0 legacy=0x7fba4415e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:26.487 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.481+0000 7fba427fc700 1 -- 172.21.15.91:0/3299297733 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba440b3ae0 con 0x7fba441440f0 2023-11-24T06:09:26.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.481+0000 7fba41ffb700 1 -- 172.21.15.91:0/3299297733 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449571415 0 0) 0x7fba44143ec0 con 0x7fba440720f0 2023-11-24T06:09:26.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.481+0000 7fba42ffd700 1 --2- 172.21.15.91:0/3299297733 >> v2:172.21.15.94:3300/0 conn(0x7fba44143ae0 0x7fba440b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:26.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.481+0000 7fba427fc700 1 --2- 172.21.15.91:0/3299297733 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fba441440f0 0x7fba44071bb0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fba4406d2e0 tx=0x7fba34009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bebfa008b24bae74 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:26.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.481+0000 7fba41ffb700 1 -- 172.21.15.91:0/3299297733 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34011830 con 0x7fba441440f0 2023-11-24T06:09:26.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.481+0000 7fba41ffb700 1 -- 172.21.15.91:0/3299297733 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba34011990 con 0x7fba441440f0 2023-11-24T06:09:26.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.481+0000 7fba41ffb700 1 -- 172.21.15.91:0/3299297733 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34011d20 con 0x7fba441440f0 2023-11-24T06:09:26.488 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.481+0000 7fba49707700 1 -- 172.21.15.91:0/3299297733 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fba441440f0 msgr2=0x7fba44071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:26.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.481+0000 7fba49707700 1 --2- 172.21.15.91:0/3299297733 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fba441440f0 0x7fba44071bb0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fba4406d2e0 tx=0x7fba34009b50 comp rx=0 tx=0).stop 2023-11-24T06:09:26.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.482+0000 7fba49707700 1 -- 172.21.15.91:0/3299297733 shutdown_connections 2023-11-24T06:09:26.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.482+0000 7fba49707700 1 --2- 172.21.15.91:0/3299297733 >> v2:172.21.15.94:3300/0 conn(0x7fba44143ae0 0x7fba440b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.482+0000 7fba49707700 1 --2- 172.21.15.91:0/3299297733 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fba441440f0 0x7fba44071bb0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.482+0000 7fba49707700 1 -- 172.21.15.91:0/3299297733 >> 172.21.15.91:0/3299297733 conn(0x7fba440b2740 msgr2=0x7fba440b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:26.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.482+0000 7fba49707700 1 -- 172.21.15.91:0/3299297733 shutdown_connections 2023-11-24T06:09:26.489 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.482+0000 7fba49707700 1 -- 172.21.15.91:0/3299297733 wait complete. 2023-11-24T06:09:26.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.483+0000 7fba49707700 1 Processor -- start 2023-11-24T06:09:26.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.483+0000 7fba49707700 1 -- start start 2023-11-24T06:09:26.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba49707700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fba440720f0 0x7fba4413acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba49707700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fba44143ae0 0x7fba4413db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba49707700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba441440f0 0x7fba44154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba49707700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4413e0b0 con 0x7fba44143ae0 2023-11-24T06:09:26.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba49707700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba441567e0 con 0x7fba441440f0 2023-11-24T06:09:26.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba49707700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba44156a30 con 0x7fba440720f0 2023-11-24T06:09:26.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba427fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fba44143ae0 0x7fba4413db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba427fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fba44143ae0 0x7fba4413db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60350/0 (socket says 172.21.15.91:60350) 2023-11-24T06:09:26.493 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba427fc700 1 -- 172.21.15.91:0/58359546 learned_addr learned my addr 172.21.15.91:0/58359546 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:26.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba42ffd700 1 --2- 172.21.15.91:0/58359546 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fba440720f0 0x7fba4413acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba437fe700 1 --2- 172.21.15.91:0/58359546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba441440f0 0x7fba44154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba427fc700 1 -- 172.21.15.91:0/58359546 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fba440720f0 msgr2=0x7fba4413acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:26.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba427fc700 1 --2- 172.21.15.91:0/58359546 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fba440720f0 0x7fba4413acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba427fc700 1 -- 172.21.15.91:0/58359546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba441440f0 msgr2=0x7fba44154420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:26.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba427fc700 1 --2- 172.21.15.91:0/58359546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba441440f0 0x7fba44154420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba427fc700 1 -- 172.21.15.91:0/58359546 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba340094a0 con 0x7fba44143ae0 2023-11-24T06:09:26.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba427fc700 1 --2- 172.21.15.91:0/58359546 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fba44143ae0 0x7fba4413db70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fba340090b0 tx=0x7fba34004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:26.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba2b7fe700 1 -- 172.21.15.91:0/58359546 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba340036a0 con 0x7fba44143ae0 2023-11-24T06:09:26.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba2b7fe700 1 -- 172.21.15.91:0/58359546 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba34003800 con 0x7fba44143ae0 2023-11-24T06:09:26.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba49707700 1 -- 172.21.15.91:0/58359546 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba44156d10 con 0x7fba44143ae0 2023-11-24T06:09:26.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.484+0000 7fba2b7fe700 1 -- 172.21.15.91:0/58359546 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34003bb0 con 0x7fba44143ae0 2023-11-24T06:09:26.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.485+0000 7fba49707700 1 -- 172.21.15.91:0/58359546 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba441572d0 con 0x7fba44143ae0 2023-11-24T06:09:26.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.485+0000 7fba49707700 1 -- 172.21.15.91:0/58359546 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba44071f60 con 0x7fba44143ae0 2023-11-24T06:09:26.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.486+0000 7fba2b7fe700 1 -- 172.21.15.91:0/58359546 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fba34003d10 con 0x7fba44143ae0 2023-11-24T06:09:26.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.486+0000 7fba2b7fe700 1 --2- 172.21.15.91:0/58359546 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fba3006af60 0x7fba3006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.486+0000 7fba2b7fe700 1 -- 172.21.15.91:0/58359546 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fba3408c690 con 0x7fba44143ae0 2023-11-24T06:09:26.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.486+0000 7fba42ffd700 1 --2- 172.21.15.91:0/58359546 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fba3006af60 0x7fba3006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.487+0000 7fba42ffd700 1 --2- 172.21.15.91:0/58359546 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fba3006af60 0x7fba3006d420 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fba38006fd0 tx=0x7fba38008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:26.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.491+0000 7fba2b7fe700 1 -- 172.21.15.91:0/58359546 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba34057c00 con 0x7fba44143ae0 2023-11-24T06:09:26.681 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.677+0000 7fba49707700 1 -- 172.21.15.91:0/58359546 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fba4415a550 con 0x7fba3006af60 2023-11-24T06:09:26.683 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.679+0000 7fba2b7fe700 1 -- 172.21.15.91:0/58359546 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50191 (secure 0 0 0) 0x7fba4415a550 con 0x7fba3006af60 2023-11-24T06:09:26.683 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:26.685 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.681+0000 7fba49707700 1 -- 172.21.15.91:0/58359546 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fba3006af60 msgr2=0x7fba3006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:26.685 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.681+0000 7fba49707700 1 --2- 172.21.15.91:0/58359546 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fba3006af60 0x7fba3006d420 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fba38006fd0 tx=0x7fba38008040 comp rx=0 tx=0).stop 2023-11-24T06:09:26.685 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.681+0000 7fba49707700 1 -- 172.21.15.91:0/58359546 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fba44143ae0 msgr2=0x7fba4413db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:26.685 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.681+0000 7fba49707700 1 --2- 172.21.15.91:0/58359546 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fba44143ae0 0x7fba4413db70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fba340090b0 tx=0x7fba34004750 comp rx=0 tx=0).stop 2023-11-24T06:09:26.686 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.681+0000 7fba49707700 1 -- 172.21.15.91:0/58359546 shutdown_connections 2023-11-24T06:09:26.686 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.681+0000 7fba49707700 1 --2- 172.21.15.91:0/58359546 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fba3006af60 0x7fba3006d420 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.686 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.681+0000 7fba49707700 1 --2- 172.21.15.91:0/58359546 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fba441440f0 0x7fba44154420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.686 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.681+0000 7fba49707700 1 --2- 172.21.15.91:0/58359546 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fba440720f0 0x7fba4413acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.686 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.681+0000 7fba49707700 1 --2- 172.21.15.91:0/58359546 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fba44143ae0 0x7fba4413db70 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.686 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.681+0000 7fba49707700 1 -- 172.21.15.91:0/58359546 >> 172.21.15.91:0/58359546 conn(0x7fba440b2740 msgr2=0x7fba440b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:26.686 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.681+0000 7fba49707700 1 -- 172.21.15.91:0/58359546 shutdown_connections 2023-11-24T06:09:26.687 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.681+0000 7fba49707700 1 -- 172.21.15.91:0/58359546 wait complete. 2023-11-24T06:09:26.687 INFO:teuthology.orchestra.run.smithi091.stderr:dumped all 2023-11-24T06:09:26.697 INFO:teuthology.orchestra.run.smithi091.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2023-11-24T06:09:25.414072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220520,"kb_used_data":4352,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754200,"statfs":{"total":773094113280,"available":772868300800,"internally_reserved":0,"allocated":4456448,"data_stored":2496614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38149,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-68,"num_write_kb":-62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001185"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047270+0000","last_change":"2023-11-24T06:07:49.463719+0000","last_active":"2023-11-24T06:09:00.047270+0000","last_peered":"2023-11-24T06:09:00.047270+0000","last_clean":"2023-11-24T06:09:00.047270+0000","last_became_active":"2023-11-24T06:07:49.462736+0000","last_became_peered":"2023-11-24T06:07:49.462736+0000","last_unstale":"2023-11-24T06:09:00.047270+0000","last_undegraded":"2023-11-24T06:09:00.047270+0000","last_fullsized":"2023-11-24T06:09:00.047270+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_clean_scrub_stamp":"2023-11-24T06:07:47.860604+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:16.508770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":983040,"data_stored":742365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.432}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.499}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.45600000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.58999999999999997}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.55000000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.69899999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.72799999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":158604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.498},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.46300000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.51500000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.41099999999999998}]},{"osd":3,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.47999999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.318},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.36099999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.52800000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.443}]}]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":438272,"data_stored":189280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.7670000000000003,"5min":4.7670000000000003,"15min":4.7670000000000003},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.0810000000000004,"5min":6.0810000000000004,"15min":6.0810000000000004},"last":0.315}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.8340000000000001,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.375}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.35499999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.9529999999999998,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.60499999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.9640000000000004,"5min":5.9640000000000004,"15min":5.9640000000000004},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":3.02,"5min":3.02,"15min":3.02},"last":0.50900000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":6.048,"5min":6.048,"15min":6.048},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.9929999999999999,"5min":2.9929999999999999,"15min":2.9929999999999999},"last":0.57799999999999996}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":6.0129999999999999,"5min":6.0129999999999999,"15min":6.0129999999999999},"last":0.52100000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":748485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":12.076000000000001,"5min":12.076000000000001,"15min":12.076000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":4.181,"5min":4.181,"15min":4.181},"last":0.39600000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.2089999999999996,"5min":4.2089999999999996,"15min":4.2089999999999996},"last":0.435}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.30399999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":4.1520000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.59799999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":4.1260000000000003,"5min":4.1260000000000003,"15min":4.1260000000000003},"last":0.52600000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.73299999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.626}]}]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":158058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.7379999999999995,"5min":9.7379999999999995,"15min":9.7379999999999995},"last":0.309}]},{"osd":1,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":11.967000000000001,"5min":11.967000000000001,"15min":11.967000000000001},"last":0.252},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":9.593,"5min":9.593,"15min":9.593},"last":0.34399999999999997}]},{"osd":3,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.373},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.32600000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":9.6479999999999997,"5min":9.6479999999999997,"15min":9.6479999999999997},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.433}]},{"osd":5,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":9.7119999999999997,"5min":9.7119999999999997,"15min":9.7119999999999997},"last":0.502}]},{"osd":6,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.52000000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":9.6859999999999999,"5min":9.6859999999999999,"15min":9.6859999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.41599999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":434176,"data_stored":189151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.41199999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.38200000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.47899999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.36099999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.439},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.63700000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.505}]},{"osd":7,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.40300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":393216,"data_stored":152011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.9539999999999997,"5min":6.9539999999999997,"15min":6.9539999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":7.0039999999999996,"5min":7.0039999999999996,"15min":7.0039999999999996},"last":0.42699999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":12.907,"5min":12.907,"15min":12.907},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":6.9710000000000001,"5min":6.9710000000000001,"15min":6.9710000000000001},"last":0.371}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.51500000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":7.0540000000000003,"5min":7.0540000000000003,"15min":7.0540000000000003},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":7.0679999999999996,"5min":7.0679999999999996,"15min":7.0679999999999996},"last":0.499}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.52700000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55500000000000005}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.54300000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":158660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.31},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.38700000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.996,"5min":1.996,"15min":1.996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.55300000000000005}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":1.9079999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.47999999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.53200000000000003}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.41499999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.497},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.70799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:09:26.698 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-24T06:09:26.699 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:09:26.820 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.816+0000 7fc67c6e9700 1 Processor -- start 2023-11-24T06:09:26.820 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.816+0000 7fc67c6e9700 1 -- start start 2023-11-24T06:09:26.822 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc67c6e9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc67415ae00 0x7fc6740b5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc67c6e9700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fc6740b03b0 0x7fc6740ad690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc67c6e9700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fc6740adbd0 0x7fc67415e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc67c6e9700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc674073610 con 0x7fc6740b03b0 2023-11-24T06:09:26.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc67c6e9700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc67415b1e0 con 0x7fc6740adbd0 2023-11-24T06:09:26.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc67c6e9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6740b6380 con 0x7fc67415ae00 2023-11-24T06:09:26.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc679c84700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fc6740b03b0 0x7fc6740ad690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.823 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc679c84700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fc6740b03b0 0x7fc6740ad690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60356/0 (socket says 172.21.15.91:60356) 2023-11-24T06:09:26.824 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc679c84700 1 -- 172.21.15.91:0/1145123817 learned_addr learned my addr 172.21.15.91:0/1145123817 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:26.824 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc67a485700 1 --2- 172.21.15.91:0/1145123817 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc67415ae00 0x7fc6740b5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.824 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc67ac86700 1 --2- 172.21.15.91:0/1145123817 >> v2:172.21.15.94:3300/0 conn(0x7fc6740adbd0 0x7fc67415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.824 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc679c84700 1 -- 172.21.15.91:0/1145123817 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc67415ae00 msgr2=0x7fc6740b5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:26.824 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc679c84700 1 --2- 172.21.15.91:0/1145123817 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc67415ae00 0x7fc6740b5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.824 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc679c84700 1 -- 172.21.15.91:0/1145123817 >> v2:172.21.15.94:3300/0 conn(0x7fc6740adbd0 msgr2=0x7fc67415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:26.824 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc679c84700 1 --2- 172.21.15.91:0/1145123817 >> v2:172.21.15.94:3300/0 conn(0x7fc6740adbd0 0x7fc67415e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.825 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc679c84700 1 -- 172.21.15.91:0/1145123817 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6740b0ec0 con 0x7fc6740b03b0 2023-11-24T06:09:26.825 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc67ac86700 1 --2- 172.21.15.91:0/1145123817 >> v2:172.21.15.94:3300/0 conn(0x7fc6740adbd0 0x7fc67415e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:26.825 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.817+0000 7fc679c84700 1 --2- 172.21.15.91:0/1145123817 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc6740b03b0 0x7fc6740ad690 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc670009920 tx=0x7fc670009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7762d2bc1468e10c server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:26.825 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.818+0000 7fc679483700 1 -- 172.21.15.91:0/1145123817 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67000b9e0 con 0x7fc6740b03b0 2023-11-24T06:09:26.825 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.818+0000 7fc679483700 1 -- 172.21.15.91:0/1145123817 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67000bb40 con 0x7fc6740b03b0 2023-11-24T06:09:26.825 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.818+0000 7fc679483700 1 -- 172.21.15.91:0/1145123817 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670009e70 con 0x7fc6740b03b0 2023-11-24T06:09:26.825 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.818+0000 7fc67c6e9700 1 -- 172.21.15.91:0/1145123817 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc6740b03b0 msgr2=0x7fc6740ad690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:26.825 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.818+0000 7fc67c6e9700 1 --2- 172.21.15.91:0/1145123817 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc6740b03b0 0x7fc6740ad690 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc670009920 tx=0x7fc670009c30 comp rx=0 tx=0).stop 2023-11-24T06:09:26.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.818+0000 7fc67c6e9700 1 -- 172.21.15.91:0/1145123817 shutdown_connections 2023-11-24T06:09:26.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.818+0000 7fc67c6e9700 1 --2- 172.21.15.91:0/1145123817 >> v2:172.21.15.94:3300/0 conn(0x7fc6740adbd0 0x7fc67415e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.818+0000 7fc67c6e9700 1 --2- 172.21.15.91:0/1145123817 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc67415ae00 0x7fc6740b5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.818+0000 7fc67c6e9700 1 --2- 172.21.15.91:0/1145123817 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc6740b03b0 0x7fc6740ad690 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.818+0000 7fc67c6e9700 1 -- 172.21.15.91:0/1145123817 >> 172.21.15.91:0/1145123817 conn(0x7fc6740b5070 msgr2=0x7fc6740b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:26.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.818+0000 7fc67c6e9700 1 -- 172.21.15.91:0/1145123817 shutdown_connections 2023-11-24T06:09:26.826 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.818+0000 7fc67c6e9700 1 -- 172.21.15.91:0/1145123817 wait complete. 2023-11-24T06:09:26.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.819+0000 7fc67c6e9700 1 Processor -- start 2023-11-24T06:09:26.829 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.819+0000 7fc67c6e9700 1 -- start start 2023-11-24T06:09:26.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.820+0000 7fc67c6e9700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6740adbd0 0x7fc6741edba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.820+0000 7fc67c6e9700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc6740b03b0 0x7fc674151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.820+0000 7fc67c6e9700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc67415ae00 0x7fc6741f1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.820+0000 7fc67c6e9700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc67414d890 con 0x7fc6740b03b0 2023-11-24T06:09:26.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.820+0000 7fc67c6e9700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6741f1540 con 0x7fc67415ae00 2023-11-24T06:09:26.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.820+0000 7fc67c6e9700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc674152460 con 0x7fc6740adbd0 2023-11-24T06:09:26.830 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.820+0000 7fc679c84700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc6740b03b0 0x7fc674151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.820+0000 7fc679c84700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc6740b03b0 0x7fc674151da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60362/0 (socket says 172.21.15.91:60362) 2023-11-24T06:09:26.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.820+0000 7fc679c84700 1 -- 172.21.15.91:0/924868176 learned_addr learned my addr 172.21.15.91:0/924868176 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:26.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.820+0000 7fc67a485700 1 --2- 172.21.15.91:0/924868176 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6740adbd0 0x7fc6741edba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.820+0000 7fc67ac86700 1 --2- 172.21.15.91:0/924868176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc67415ae00 0x7fc6741f1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.821+0000 7fc679c84700 1 -- 172.21.15.91:0/924868176 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6740adbd0 msgr2=0x7fc6741edba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:26.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.821+0000 7fc679c84700 1 --2- 172.21.15.91:0/924868176 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6740adbd0 0x7fc6741edba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.821+0000 7fc679c84700 1 -- 172.21.15.91:0/924868176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc67415ae00 msgr2=0x7fc6741f1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:26.831 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.821+0000 7fc679c84700 1 --2- 172.21.15.91:0/924868176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc67415ae00 0x7fc6741f1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:26.832 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.821+0000 7fc679c84700 1 -- 172.21.15.91:0/924868176 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc670009550 con 0x7fc6740b03b0 2023-11-24T06:09:26.832 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.821+0000 7fc67ac86700 1 --2- 172.21.15.91:0/924868176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc67415ae00 0x7fc6741f1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:26.832 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.821+0000 7fc679c84700 1 --2- 172.21.15.91:0/924868176 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc6740b03b0 0x7fc674151da0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc670009c70 tx=0x7fc670004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:26.832 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.821+0000 7fc66affd700 1 -- 172.21.15.91:0/924868176 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670004510 con 0x7fc6740b03b0 2023-11-24T06:09:26.832 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.821+0000 7fc66affd700 1 -- 172.21.15.91:0/924868176 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6700036a0 con 0x7fc6740b03b0 2023-11-24T06:09:26.832 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.821+0000 7fc66affd700 1 -- 172.21.15.91:0/924868176 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670003a50 con 0x7fc6740b03b0 2023-11-24T06:09:26.832 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.821+0000 7fc67c6e9700 1 -- 172.21.15.91:0/924868176 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc674152740 con 0x7fc6740b03b0 2023-11-24T06:09:26.832 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.821+0000 7fc67c6e9700 1 -- 172.21.15.91:0/924868176 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc67414ddc0 con 0x7fc6740b03b0 2023-11-24T06:09:26.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.822+0000 7fc67c6e9700 1 -- 172.21.15.91:0/924868176 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc674068670 con 0x7fc6740b03b0 2023-11-24T06:09:26.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.822+0000 7fc66affd700 1 -- 172.21.15.91:0/924868176 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fc670003bb0 con 0x7fc6740b03b0 2023-11-24T06:09:26.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.823+0000 7fc66affd700 1 --2- 172.21.15.91:0/924868176 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc65806af60 0x7fc65806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:26.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.823+0000 7fc66affd700 1 -- 172.21.15.91:0/924868176 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc67008bfe0 con 0x7fc6740b03b0 2023-11-24T06:09:26.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.823+0000 7fc67a485700 1 --2- 172.21.15.91:0/924868176 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc65806af60 0x7fc65806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:26.833 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.823+0000 7fc67a485700 1 --2- 172.21.15.91:0/924868176 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc65806af60 0x7fc65806d420 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc6740734f0 tx=0x7fc664008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:26.834 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:26.827+0000 7fc66affd700 1 -- 172.21.15.91:0/924868176 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6700573b0 con 0x7fc6740b03b0 2023-11-24T06:09:27.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.013+0000 7fc67c6e9700 1 -- 172.21.15.91:0/924868176 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc67406b170 con 0x7fc65806af60 2023-11-24T06:09:27.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.015+0000 7fc66affd700 1 -- 172.21.15.91:0/924868176 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50191 (secure 0 0 0) 0x7fc664007650 con 0x7fc65806af60 2023-11-24T06:09:27.020 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:27.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.017+0000 7fc67c6e9700 1 -- 172.21.15.91:0/924868176 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc65806af60 msgr2=0x7fc65806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:27.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.017+0000 7fc67c6e9700 1 --2- 172.21.15.91:0/924868176 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc65806af60 0x7fc65806d420 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc6740734f0 tx=0x7fc664008040 comp rx=0 tx=0).stop 2023-11-24T06:09:27.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.018+0000 7fc67c6e9700 1 -- 172.21.15.91:0/924868176 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc6740b03b0 msgr2=0x7fc674151da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:27.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.018+0000 7fc67c6e9700 1 --2- 172.21.15.91:0/924868176 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc6740b03b0 0x7fc674151da0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc670009c70 tx=0x7fc670004750 comp rx=0 tx=0).stop 2023-11-24T06:09:27.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.018+0000 7fc67c6e9700 1 -- 172.21.15.91:0/924868176 shutdown_connections 2023-11-24T06:09:27.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.018+0000 7fc67c6e9700 1 --2- 172.21.15.91:0/924868176 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fc65806af60 0x7fc65806d420 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:27.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.018+0000 7fc67c6e9700 1 --2- 172.21.15.91:0/924868176 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fc67415ae00 0x7fc6741f1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:27.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.018+0000 7fc67c6e9700 1 --2- 172.21.15.91:0/924868176 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fc6740adbd0 0x7fc6741edba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:27.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.018+0000 7fc67c6e9700 1 --2- 172.21.15.91:0/924868176 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fc6740b03b0 0x7fc674151da0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:27.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.018+0000 7fc67c6e9700 1 -- 172.21.15.91:0/924868176 >> 172.21.15.91:0/924868176 conn(0x7fc6740b5070 msgr2=0x7fc674154390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:27.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.018+0000 7fc67c6e9700 1 -- 172.21.15.91:0/924868176 shutdown_connections 2023-11-24T06:09:27.023 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.018+0000 7fc67c6e9700 1 -- 172.21.15.91:0/924868176 wait complete. 2023-11-24T06:09:27.023 INFO:teuthology.orchestra.run.smithi091.stderr:dumped all 2023-11-24T06:09:27.033 INFO:teuthology.orchestra.run.smithi091.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2023-11-24T06:09:25.414072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220520,"kb_used_data":4352,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754200,"statfs":{"total":773094113280,"available":772868300800,"internally_reserved":0,"allocated":4456448,"data_stored":2496614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38149,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-68,"num_write_kb":-62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001185"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-11-24T06:09:00.047270+0000","last_change":"2023-11-24T06:07:49.463719+0000","last_active":"2023-11-24T06:09:00.047270+0000","last_peered":"2023-11-24T06:09:00.047270+0000","last_clean":"2023-11-24T06:09:00.047270+0000","last_became_active":"2023-11-24T06:07:49.462736+0000","last_became_peered":"2023-11-24T06:07:49.462736+0000","last_unstale":"2023-11-24T06:09:00.047270+0000","last_undegraded":"2023-11-24T06:09:00.047270+0000","last_fullsized":"2023-11-24T06:09:00.047270+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_clean_scrub_stamp":"2023-11-24T06:07:47.860604+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:16.508770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":983040,"data_stored":742365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.432}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.499}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.45600000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.58999999999999997}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.55000000000000004}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.69899999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.72799999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":158604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.498},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.46300000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.51500000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.41099999999999998}]},{"osd":3,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.47999999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.318},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.36099999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.52800000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.443}]}]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":438272,"data_stored":189280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.7670000000000003,"5min":4.7670000000000003,"15min":4.7670000000000003},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.0810000000000004,"5min":6.0810000000000004,"15min":6.0810000000000004},"last":0.315}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.8340000000000001,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.375}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.35499999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.9529999999999998,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.60499999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.9640000000000004,"5min":5.9640000000000004,"15min":5.9640000000000004},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":3.02,"5min":3.02,"15min":3.02},"last":0.50900000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":6.048,"5min":6.048,"15min":6.048},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.9929999999999999,"5min":2.9929999999999999,"15min":2.9929999999999999},"last":0.57799999999999996}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":6.0129999999999999,"5min":6.0129999999999999,"15min":6.0129999999999999},"last":0.52100000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":748485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":12.076000000000001,"5min":12.076000000000001,"15min":12.076000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":4.181,"5min":4.181,"15min":4.181},"last":0.39600000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.2089999999999996,"5min":4.2089999999999996,"15min":4.2089999999999996},"last":0.435}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.30399999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":4.1520000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.59799999999999998}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":4.1260000000000003,"5min":4.1260000000000003,"15min":4.1260000000000003},"last":0.52600000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.73299999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.626}]}]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":158058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.7379999999999995,"5min":9.7379999999999995,"15min":9.7379999999999995},"last":0.309}]},{"osd":1,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":11.967000000000001,"5min":11.967000000000001,"15min":11.967000000000001},"last":0.252},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":9.593,"5min":9.593,"15min":9.593},"last":0.34399999999999997}]},{"osd":3,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.373},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.32600000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":9.6479999999999997,"5min":9.6479999999999997,"15min":9.6479999999999997},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.433}]},{"osd":5,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":9.7119999999999997,"5min":9.7119999999999997,"15min":9.7119999999999997},"last":0.502}]},{"osd":6,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.52000000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":9.6859999999999999,"5min":9.6859999999999999,"15min":9.6859999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.41599999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":434176,"data_stored":189151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.41199999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.38200000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.47899999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.36099999999999999}]},{"osd":5,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.439},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.63700000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.505}]},{"osd":7,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.40300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":393216,"data_stored":152011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.9539999999999997,"5min":6.9539999999999997,"15min":6.9539999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":7.0039999999999996,"5min":7.0039999999999996,"15min":7.0039999999999996},"last":0.42699999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":12.907,"5min":12.907,"15min":12.907},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":6.9710000000000001,"5min":6.9710000000000001,"15min":6.9710000000000001},"last":0.371}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.51500000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":7.0540000000000003,"5min":7.0540000000000003,"15min":7.0540000000000003},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":7.0679999999999996,"5min":7.0679999999999996,"15min":7.0679999999999996},"last":0.499}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.52700000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55500000000000005}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.54300000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":405504,"data_stored":158660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.31},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.38700000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.996,"5min":1.996,"15min":1.996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.55300000000000005}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":1.9079999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.47999999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.53200000000000003}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.41499999999999998}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.497},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.70799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:09:27.035 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-24T06:09:27.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.152+0000 7f55e1748700 1 Processor -- start 2023-11-24T06:09:27.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.152+0000 7f55e1748700 1 -- start start 2023-11-24T06:09:27.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.152+0000 7f55e1748700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f55dc0caae0 0x7f55dc0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:27.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.152+0000 7f55e1748700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55dc0d0b80 0x7f55dc0b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:27.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.152+0000 7f55e1748700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55dc0b3c40 con 0x7f55dc161900 2023-11-24T06:09:27.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.152+0000 7f55e1748700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55dc0b36e0 con 0x7f55dc0caae0 2023-11-24T06:09:27.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.152+0000 7f55e1748700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0cfbc0 con 0x7f55dc0d0b80 2023-11-24T06:09:27.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.153+0000 7f55daffd700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f55dc161900 0x7f55dc0d0470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55304/0 (socket says 172.21.15.91:55304) 2023-11-24T06:09:27.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.153+0000 7f55daffd700 1 -- 172.21.15.91:0/2741089693 learned_addr learned my addr 172.21.15.91:0/2741089693 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:27.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.153+0000 7f55db7fe700 1 --2- 172.21.15.91:0/2741089693 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55dc0d0b80 0x7f55dc0b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:27.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.153+0000 7f55da7fc700 1 --2- 172.21.15.91:0/2741089693 >> v2:172.21.15.94:3300/0 conn(0x7f55dc0caae0 0x7f55dc0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:27.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.153+0000 7f55d9ffb700 1 -- 172.21.15.91:0/2741089693 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534644630 0 0) 0x7f55dc0b3c40 con 0x7f55dc161900 2023-11-24T06:09:27.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.153+0000 7f55d9ffb700 1 -- 172.21.15.91:0/2741089693 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c40035b0 con 0x7f55dc161900 2023-11-24T06:09:27.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.153+0000 7f55da7fc700 1 -- 172.21.15.91:0/2741089693 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55dc0d0b80 msgr2=0x7f55dc0b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:27.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.153+0000 7f55da7fc700 1 --2- 172.21.15.91:0/2741089693 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55dc0d0b80 0x7f55dc0b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:27.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.153+0000 7f55da7fc700 1 -- 172.21.15.91:0/2741089693 >> v1:172.21.15.91:6789/0 conn(0x7f55dc161900 legacy=0x7f55dc0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:27.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.153+0000 7f55d9ffb700 1 -- 172.21.15.91:0/2741089693 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2762719380 0 0) 0x7f55c40035b0 con 0x7f55dc161900 2023-11-24T06:09:27.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.153+0000 7f55da7fc700 1 -- 172.21.15.91:0/2741089693 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc0cfe60 con 0x7f55dc0caae0 2023-11-24T06:09:27.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.153+0000 7f55da7fc700 1 --2- 172.21.15.91:0/2741089693 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55dc0caae0 0x7f55dc0ceb00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f55cc009840 tx=0x7f55cc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4de1da4295fa7535 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:27.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.153+0000 7f55d9ffb700 1 -- 172.21.15.91:0/2741089693 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc011830 con 0x7f55dc0caae0 2023-11-24T06:09:27.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.153+0000 7f55d9ffb700 1 -- 172.21.15.91:0/2741089693 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc011990 con 0x7f55dc0caae0 2023-11-24T06:09:27.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.153+0000 7f55d9ffb700 1 -- 172.21.15.91:0/2741089693 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc011d00 con 0x7f55dc0caae0 2023-11-24T06:09:27.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.154+0000 7f55e1748700 1 -- 172.21.15.91:0/2741089693 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55dc0caae0 msgr2=0x7f55dc0ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:27.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.154+0000 7f55e1748700 1 --2- 172.21.15.91:0/2741089693 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55dc0caae0 0x7f55dc0ceb00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f55cc009840 tx=0x7f55cc009b50 comp rx=0 tx=0).stop 2023-11-24T06:09:27.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.154+0000 7f55e1748700 1 -- 172.21.15.91:0/2741089693 shutdown_connections 2023-11-24T06:09:27.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.154+0000 7f55e1748700 1 --2- 172.21.15.91:0/2741089693 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55dc0d0b80 0x7f55dc0b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:27.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.154+0000 7f55e1748700 1 --2- 172.21.15.91:0/2741089693 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55dc0caae0 0x7f55dc0ceb00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:27.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.154+0000 7f55e1748700 1 -- 172.21.15.91:0/2741089693 >> 172.21.15.91:0/2741089693 conn(0x7f55dc0d3e40 msgr2=0x7f55dc0d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:27.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.154+0000 7f55e1748700 1 -- 172.21.15.91:0/2741089693 shutdown_connections 2023-11-24T06:09:27.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.154+0000 7f55e1748700 1 -- 172.21.15.91:0/2741089693 wait complete. 2023-11-24T06:09:27.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.155+0000 7f55e1748700 1 Processor -- start 2023-11-24T06:09:27.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.155+0000 7f55e1748700 1 -- start start 2023-11-24T06:09:27.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55e1748700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55dc0caae0 0x7f55dc0cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:27.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55e1748700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55dc0d0b80 0x7f55dc0c93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:27.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55e1748700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55dc161900 0x7f55dc1f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:27.169 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55e1748700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0c89c0 con 0x7f55dc0caae0 2023-11-24T06:09:27.169 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55e1748700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0c8f40 con 0x7f55dc0d0b80 2023-11-24T06:09:27.169 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55e1748700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0cd9b0 con 0x7f55dc161900 2023-11-24T06:09:27.169 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55daffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55dc0caae0 0x7f55dc0cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:27.169 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55daffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55dc0caae0 0x7f55dc0cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60376/0 (socket says 172.21.15.91:60376) 2023-11-24T06:09:27.170 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55daffd700 1 -- 172.21.15.91:0/371614678 learned_addr learned my addr 172.21.15.91:0/371614678 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:27.170 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55db7fe700 1 --2- 172.21.15.91:0/371614678 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55dc161900 0x7f55dc1f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:27.170 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55daffd700 1 -- 172.21.15.91:0/371614678 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55dc161900 msgr2=0x7f55dc1f3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:27.170 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55daffd700 1 --2- 172.21.15.91:0/371614678 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55dc161900 0x7f55dc1f3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:27.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55daffd700 1 -- 172.21.15.91:0/371614678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55dc0d0b80 msgr2=0x7f55dc0c93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:27.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55da7fc700 1 --2- 172.21.15.91:0/371614678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55dc0d0b80 0x7f55dc0c93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:27.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55daffd700 1 --2- 172.21.15.91:0/371614678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55dc0d0b80 0x7f55dc0c93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:27.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55daffd700 1 -- 172.21.15.91:0/371614678 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc0094a0 con 0x7f55dc0caae0 2023-11-24T06:09:27.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.156+0000 7f55da7fc700 1 --2- 172.21.15.91:0/371614678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55dc0d0b80 0x7f55dc0c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:27.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.157+0000 7f55daffd700 1 --2- 172.21.15.91:0/371614678 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55dc0caae0 0x7f55dc0cd470 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f55d000c4a0 tx=0x7f55d000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:27.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.157+0000 7f55c37fe700 1 -- 172.21.15.91:0/371614678 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d0005d70 con 0x7f55dc0caae0 2023-11-24T06:09:27.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.157+0000 7f55e1748700 1 -- 172.21.15.91:0/371614678 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55dc1f47a0 con 0x7f55dc0caae0 2023-11-24T06:09:27.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.157+0000 7f55c37fe700 1 -- 172.21.15.91:0/371614678 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d00070d0 con 0x7f55dc0caae0 2023-11-24T06:09:27.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.157+0000 7f55e1748700 1 -- 172.21.15.91:0/371614678 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55dc0c8bd0 con 0x7f55dc0caae0 2023-11-24T06:09:27.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.157+0000 7f55c37fe700 1 -- 172.21.15.91:0/371614678 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d0007440 con 0x7f55dc0caae0 2023-11-24T06:09:27.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.158+0000 7f55e1748700 1 -- 172.21.15.91:0/371614678 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55bc0052e0 con 0x7f55dc0caae0 2023-11-24T06:09:27.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.158+0000 7f55c37fe700 1 -- 172.21.15.91:0/371614678 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f55d00075a0 con 0x7f55dc0caae0 2023-11-24T06:09:27.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.158+0000 7f55c37fe700 1 --2- 172.21.15.91:0/371614678 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f55c806af60 0x7f55c806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:27.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.159+0000 7f55c37fe700 1 -- 172.21.15.91:0/371614678 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f55d0088630 con 0x7f55dc0caae0 2023-11-24T06:09:27.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.159+0000 7f55da7fc700 1 --2- 172.21.15.91:0/371614678 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f55c806af60 0x7f55c806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:27.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.159+0000 7f55da7fc700 1 --2- 172.21.15.91:0/371614678 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f55c806af60 0x7f55c806d420 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f55dc15d260 tx=0x7f55cc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:27.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.163+0000 7f55c37fe700 1 -- 172.21.15.91:0/371614678 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f55d0053ab0 con 0x7f55dc0caae0 2023-11-24T06:09:27.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.355+0000 7f55e1748700 1 -- 172.21.15.91:0/371614678 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55bc005860 con 0x7f55dc0caae0 2023-11-24T06:09:27.360 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.356+0000 7f55c37fe700 1 -- 172.21.15.91:0/371614678 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+12902 (secure 0 0 0) 0x7f55d00570d0 con 0x7f55dc0caae0 2023-11-24T06:09:27.360 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:27.361 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":37,"fsid":"f7c07073-6cd6-496f-812a-c57c86a2fbbb","created":"2023-11-24T06:07:35.801216+0000","modified":"2023-11-24T06:09:23.984163+0000","last_up_change":"2023-11-24T06:07:46.849047+0000","last_in_change":"2023-11-24T06:07:38.279996+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-24T06:07:47.435506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5296abf7-43c1-435b-87fc-71b8cf881bc5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6816","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6819","nonce":420345853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6822","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6824","nonce":420345853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6830","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6831","nonce":420345853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6826","nonce":420345853},{"type":"v1","addr":"172.21.15.91:6828","nonce":420345853}]},"public_addr":"172.21.15.91:6819/420345853","cluster_addr":"172.21.15.91:6824/420345853","heartbeat_back_addr":"172.21.15.91:6831/420345853","heartbeat_front_addr":"172.21.15.91:6828/420345853","state":["exists","up"]},{"osd":1,"uuid":"fc021485-f189-4918-b6e8-bef2e064a46b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6813","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6815","nonce":66406286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6818","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6821","nonce":66406286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6827","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6829","nonce":66406286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6823","nonce":66406286},{"type":"v1","addr":"172.21.15.91:6825","nonce":66406286}]},"public_addr":"172.21.15.91:6815/66406286","cluster_addr":"172.21.15.91:6821/66406286","heartbeat_back_addr":"172.21.15.91:6829/66406286","heartbeat_front_addr":"172.21.15.91:6825/66406286","state":["exists","up"]},{"osd":2,"uuid":"6938ed71-88b6-47e4-9c1d-fbc2856d74f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6808","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6809","nonce":649116497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6810","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6811","nonce":649116497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6817","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6820","nonce":649116497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6812","nonce":649116497},{"type":"v1","addr":"172.21.15.91:6814","nonce":649116497}]},"public_addr":"172.21.15.91:6809/649116497","cluster_addr":"172.21.15.91:6811/649116497","heartbeat_back_addr":"172.21.15.91:6820/649116497","heartbeat_front_addr":"172.21.15.91:6814/649116497","state":["exists","up"]},{"osd":3,"uuid":"07a912f1-95b0-4426-85d6-61f77bfa99bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6800","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6801","nonce":1071490516}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6802","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6803","nonce":1071490516}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6806","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6807","nonce":1071490516}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6804","nonce":1071490516},{"type":"v1","addr":"172.21.15.91:6805","nonce":1071490516}]},"public_addr":"172.21.15.91:6801/1071490516","cluster_addr":"172.21.15.91:6803/1071490516","heartbeat_back_addr":"172.21.15.91:6807/1071490516","heartbeat_front_addr":"172.21.15.91:6805/1071490516","state":["exists","up"]},{"osd":4,"uuid":"5f1c8835-4980-4ced-8b44-fb6bebe8a777","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6824","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6825","nonce":3888015416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6826","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6827","nonce":3888015416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6830","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6831","nonce":3888015416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6828","nonce":3888015416},{"type":"v1","addr":"172.21.15.94:6829","nonce":3888015416}]},"public_addr":"172.21.15.94:6825/3888015416","cluster_addr":"172.21.15.94:6827/3888015416","heartbeat_back_addr":"172.21.15.94:6831/3888015416","heartbeat_front_addr":"172.21.15.94:6829/3888015416","state":["exists","up"]},{"osd":5,"uuid":"de892401-7f4e-423f-9447-4a750970534f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6808","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6809","nonce":4188708142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6810","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6811","nonce":4188708142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6814","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6815","nonce":4188708142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6812","nonce":4188708142},{"type":"v1","addr":"172.21.15.94:6813","nonce":4188708142}]},"public_addr":"172.21.15.94:6809/4188708142","cluster_addr":"172.21.15.94:6811/4188708142","heartbeat_back_addr":"172.21.15.94:6815/4188708142","heartbeat_front_addr":"172.21.15.94:6813/4188708142","state":["exists","up"]},{"osd":6,"uuid":"9c4c63c3-0897-43c4-8ab6-d9c8a50b8f71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6800","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6801","nonce":4250977597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6802","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6803","nonce":4250977597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6806","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6807","nonce":4250977597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6804","nonce":4250977597},{"type":"v1","addr":"172.21.15.94:6805","nonce":4250977597}]},"public_addr":"172.21.15.94:6801/4250977597","cluster_addr":"172.21.15.94:6803/4250977597","heartbeat_back_addr":"172.21.15.94:6807/4250977597","heartbeat_front_addr":"172.21.15.94:6805/4250977597","state":["exists","up"]},{"osd":7,"uuid":"786a3f34-eb17-4f18-8f32-a903286f0a2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6816","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6817","nonce":3673570051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6818","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6819","nonce":3673570051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6822","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6823","nonce":3673570051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6820","nonce":3673570051},{"type":"v1","addr":"172.21.15.94:6821","nonce":3673570051}]},"public_addr":"172.21.15.94:6817/3673570051","cluster_addr":"172.21.15.94:6819/3673570051","heartbeat_back_addr":"172.21.15.94:6823/3673570051","heartbeat_front_addr":"172.21.15.94:6821/3673570051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.502843+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.461319+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.477531+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.490372+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.400854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.478841+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.445905+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-11-24T06:07:43.422566+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.91:6837/1301930708":"2023-11-25T06:09:20.975296+0000","172.21.15.91:6835/1301930708":"2023-11-25T06:09:20.975296+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-24T06:09:27.362 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.358+0000 7f55e1748700 1 -- 172.21.15.91:0/371614678 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f55c806af60 msgr2=0x7f55c806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:27.363 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.358+0000 7f55e1748700 1 --2- 172.21.15.91:0/371614678 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f55c806af60 0x7f55c806d420 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f55dc15d260 tx=0x7f55cc008030 comp rx=0 tx=0).stop 2023-11-24T06:09:27.363 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.358+0000 7f55e1748700 1 -- 172.21.15.91:0/371614678 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55dc0caae0 msgr2=0x7f55dc0cd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:27.363 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.358+0000 7f55e1748700 1 --2- 172.21.15.91:0/371614678 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55dc0caae0 0x7f55dc0cd470 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f55d000c4a0 tx=0x7f55d000c860 comp rx=0 tx=0).stop 2023-11-24T06:09:27.363 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.358+0000 7f55e1748700 1 -- 172.21.15.91:0/371614678 shutdown_connections 2023-11-24T06:09:27.363 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.358+0000 7f55e1748700 1 --2- 172.21.15.91:0/371614678 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f55c806af60 0x7f55c806d420 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:27.363 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.358+0000 7f55e1748700 1 --2- 172.21.15.91:0/371614678 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f55dc161900 0x7f55dc1f3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:27.364 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.358+0000 7f55e1748700 1 --2- 172.21.15.91:0/371614678 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f55dc0caae0 0x7f55dc0cd470 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:27.364 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.358+0000 7f55e1748700 1 --2- 172.21.15.91:0/371614678 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f55dc0d0b80 0x7f55dc0c93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:27.364 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.358+0000 7f55e1748700 1 -- 172.21.15.91:0/371614678 >> 172.21.15.91:0/371614678 conn(0x7f55dc0d3e40 msgr2=0x7f55dc15e430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:27.364 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.358+0000 7f55e1748700 1 -- 172.21.15.91:0/371614678 shutdown_connections 2023-11-24T06:09:27.364 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:27.358+0000 7f55e1748700 1 -- 172.21.15.91:0/371614678 wait complete. 2023-11-24T06:09:28.375 INFO:tasks.ceph:Scrubbing osd.0 2023-11-24T06:09:28.376 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-11-24T06:09:28.505 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.502+0000 7f53f6fcb700 1 Processor -- start 2023-11-24T06:09:28.506 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.502+0000 7f53f6fcb700 1 -- start start 2023-11-24T06:09:28.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.502+0000 7f53f6fcb700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f53f01457b0 0x7f53f00725a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:28.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.502+0000 7f53f6fcb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53f006f580 0x7f53f006e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:28.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.502+0000 7f53f6fcb700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f00cd490 con 0x7f53f006e910 2023-11-24T06:09:28.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.502+0000 7f53f6fcb700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f006fa00 con 0x7f53f01457b0 2023-11-24T06:09:28.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.502+0000 7f53f6fcb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f00cd6e0 con 0x7f53f006f580 2023-11-24T06:09:28.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.503+0000 7f53f4d67700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f53f01457b0 0x7f53f00725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:28.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.503+0000 7f53f5568700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f53f006e910 0x7f53f00cf020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55322/0 (socket says 172.21.15.91:55322) 2023-11-24T06:09:28.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.503+0000 7f53f5568700 1 -- 172.21.15.91:0/2637998390 learned_addr learned my addr 172.21.15.91:0/2637998390 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:28.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.503+0000 7f53f4d67700 1 -- 172.21.15.91:0/2637998390 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53f006f580 msgr2=0x7f53f006e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:28.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.503+0000 7f53effff700 1 --2- 172.21.15.91:0/2637998390 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53f006f580 0x7f53f006e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:28.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.503+0000 7f53f4d67700 1 --2- 172.21.15.91:0/2637998390 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53f006f580 0x7f53f006e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:28.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.503+0000 7f53f4d67700 1 -- 172.21.15.91:0/2637998390 >> v1:172.21.15.91:6789/0 conn(0x7f53f006e910 legacy=0x7f53f00cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:28.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.503+0000 7f53ef7fe700 1 -- 172.21.15.91:0/2637998390 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833406493 0 0) 0x7f53f00cd490 con 0x7f53f006e910 2023-11-24T06:09:28.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.503+0000 7f53f4d67700 1 -- 172.21.15.91:0/2637998390 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f00cd9c0 con 0x7f53f01457b0 2023-11-24T06:09:28.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.503+0000 7f53f4d67700 1 --2- 172.21.15.91:0/2637998390 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53f01457b0 0x7f53f00725a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f53e4009840 tx=0x7f53e4009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29451b954b17e5b3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:28.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.503+0000 7f53ef7fe700 1 -- 172.21.15.91:0/2637998390 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e4011830 con 0x7f53f01457b0 2023-11-24T06:09:28.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.503+0000 7f53ef7fe700 1 -- 172.21.15.91:0/2637998390 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e4011990 con 0x7f53f01457b0 2023-11-24T06:09:28.511 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.503+0000 7f53ef7fe700 1 -- 172.21.15.91:0/2637998390 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e4011d20 con 0x7f53f01457b0 2023-11-24T06:09:28.511 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.504+0000 7f53f6fcb700 1 -- 172.21.15.91:0/2637998390 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53f01457b0 msgr2=0x7f53f00725a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:28.511 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.504+0000 7f53f6fcb700 1 --2- 172.21.15.91:0/2637998390 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53f01457b0 0x7f53f00725a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f53e4009840 tx=0x7f53e4009b50 comp rx=0 tx=0).stop 2023-11-24T06:09:28.511 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.504+0000 7f53f6fcb700 1 -- 172.21.15.91:0/2637998390 shutdown_connections 2023-11-24T06:09:28.511 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.504+0000 7f53f6fcb700 1 --2- 172.21.15.91:0/2637998390 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53f01457b0 0x7f53f00725a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:28.511 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.504+0000 7f53f6fcb700 1 --2- 172.21.15.91:0/2637998390 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53f006f580 0x7f53f006e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:28.511 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.504+0000 7f53f6fcb700 1 -- 172.21.15.91:0/2637998390 >> 172.21.15.91:0/2637998390 conn(0x7f53f00b2760 msgr2=0x7f53f00b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:28.512 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.504+0000 7f53f6fcb700 1 -- 172.21.15.91:0/2637998390 shutdown_connections 2023-11-24T06:09:28.512 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.504+0000 7f53f6fcb700 1 -- 172.21.15.91:0/2637998390 wait complete. 2023-11-24T06:09:28.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.505+0000 7f53f6fcb700 1 Processor -- start 2023-11-24T06:09:28.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.505+0000 7f53f6fcb700 1 -- start start 2023-11-24T06:09:28.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53f6fcb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53f006e910 0x7f53f013c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:28.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53f6fcb700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53f006f580 0x7f53f00cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:28.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53f6fcb700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53f01457b0 0x7f53f015eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:28.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53f6fcb700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f015c9c0 con 0x7f53f01457b0 2023-11-24T06:09:28.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53f6fcb700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f013c700 con 0x7f53f006f580 2023-11-24T06:09:28.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53f6fcb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f013c9e0 con 0x7f53f006e910 2023-11-24T06:09:28.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53f5568700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53f01457b0 0x7f53f015eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:28.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53f4d67700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53f006e910 0x7f53f013c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:28.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53f5568700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53f01457b0 0x7f53f015eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60394/0 (socket says 172.21.15.91:60394) 2023-11-24T06:09:28.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53f5568700 1 -- 172.21.15.91:0/1916193200 learned_addr learned my addr 172.21.15.91:0/1916193200 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:28.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53effff700 1 --2- 172.21.15.91:0/1916193200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53f006f580 0x7f53f00cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:28.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53f5568700 1 -- 172.21.15.91:0/1916193200 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53f006e910 msgr2=0x7f53f013c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:28.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53f5568700 1 --2- 172.21.15.91:0/1916193200 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f53f006e910 0x7f53f013c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:28.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53f5568700 1 -- 172.21.15.91:0/1916193200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53f006f580 msgr2=0x7f53f00cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:28.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53f5568700 1 --2- 172.21.15.91:0/1916193200 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f53f006f580 0x7f53f00cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:28.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53f5568700 1 -- 172.21.15.91:0/1916193200 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e40094a0 con 0x7f53f01457b0 2023-11-24T06:09:28.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.506+0000 7f53f5568700 1 --2- 172.21.15.91:0/1916193200 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53f01457b0 0x7f53f015eef0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f53e800cca0 tx=0x7f53e800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:28.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.507+0000 7f53ed7fa700 1 -- 172.21.15.91:0/1916193200 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e800ab70 con 0x7f53f01457b0 2023-11-24T06:09:28.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.507+0000 7f53ed7fa700 1 -- 172.21.15.91:0/1916193200 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e800acd0 con 0x7f53f01457b0 2023-11-24T06:09:28.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.507+0000 7f53f6fcb700 1 -- 172.21.15.91:0/1916193200 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f015c370 con 0x7f53f01457b0 2023-11-24T06:09:28.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.507+0000 7f53ed7fa700 1 -- 172.21.15.91:0/1916193200 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e8007310 con 0x7f53f01457b0 2023-11-24T06:09:28.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.507+0000 7f53f6fcb700 1 -- 172.21.15.91:0/1916193200 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f015ce50 con 0x7f53f01457b0 2023-11-24T06:09:28.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.507+0000 7f53f6fcb700 1 -- 172.21.15.91:0/1916193200 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f53f006deb0 con 0x7f53f01457b0 2023-11-24T06:09:28.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.508+0000 7f53ed7fa700 1 -- 172.21.15.91:0/1916193200 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f53e8007470 con 0x7f53f01457b0 2023-11-24T06:09:28.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.508+0000 7f53ed7fa700 1 --2- 172.21.15.91:0/1916193200 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f53d406af60 0x7f53d406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:28.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.508+0000 7f53ed7fa700 1 -- 172.21.15.91:0/1916193200 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f53e8088df0 con 0x7f53f01457b0 2023-11-24T06:09:28.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.508+0000 7f53f4d67700 1 --2- 172.21.15.91:0/1916193200 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f53d406af60 0x7f53d406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:28.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.509+0000 7f53ed7fa700 1 --2- 172.21.15.91:0/1916193200 >> [v2:172.21.15.91:6816/420345853,v1:172.21.15.91:6819/420345853] conn(0x7f53d4070dc0 0x7f53d40731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:28.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.509+0000 7f53ed7fa700 1 -- 172.21.15.91:0/1916193200 --> [v2:172.21.15.91:6816/420345853,v1:172.21.15.91:6819/420345853] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f53d40738b0 con 0x7f53d4070dc0 2023-11-24T06:09:28.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.509+0000 7f53ed7fa700 1 -- 172.21.15.91:0/1916193200 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53e808b500 con 0x7f53f01457b0 2023-11-24T06:09:28.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.509+0000 7f53effff700 1 --2- 172.21.15.91:0/1916193200 >> [v2:172.21.15.91:6816/420345853,v1:172.21.15.91:6819/420345853] conn(0x7f53d4070dc0 0x7f53d40731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:28.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.509+0000 7f53f4d67700 1 --2- 172.21.15.91:0/1916193200 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f53d406af60 0x7f53d406d420 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f53e40090b0 tx=0x7f53e401c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:28.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.509+0000 7f53effff700 1 --2- 172.21.15.91:0/1916193200 >> [v2:172.21.15.91:6816/420345853,v1:172.21.15.91:6819/420345853] conn(0x7f53d4070dc0 0x7f53d40731e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:28.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.510+0000 7f53ed7fa700 1 -- 172.21.15.91:0/1916193200 <== osd.0 v2:172.21.15.91:6816/420345853 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f53d40738b0 con 0x7f53d4070dc0 2023-11-24T06:09:28.528 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.524+0000 7f53f6fcb700 1 -- 172.21.15.91:0/1916193200 --> [v2:172.21.15.91:6816/420345853,v1:172.21.15.91:6819/420345853] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f53f013cc00 con 0x7f53d4070dc0 2023-11-24T06:09:28.528 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.525+0000 7f53ed7fa700 1 -- 172.21.15.91:0/1916193200 <== osd.0 v2:172.21.15.91:6816/420345853 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f53f013cc00 con 0x7f53d4070dc0 2023-11-24T06:09:28.529 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2023-11-24T06:09:28.529 INFO:teuthology.orchestra.run.smithi091.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T06:09:28.529 INFO:teuthology.orchestra.run.smithi091.stdout:} 2023-11-24T06:09:28.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.525+0000 7f53f6fcb700 1 -- 172.21.15.91:0/1916193200 >> [v2:172.21.15.91:6816/420345853,v1:172.21.15.91:6819/420345853] conn(0x7f53d4070dc0 msgr2=0x7f53d40731e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:28.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.525+0000 7f53f6fcb700 1 --2- 172.21.15.91:0/1916193200 >> [v2:172.21.15.91:6816/420345853,v1:172.21.15.91:6819/420345853] conn(0x7f53d4070dc0 0x7f53d40731e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:28.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.525+0000 7f53f6fcb700 1 -- 172.21.15.91:0/1916193200 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f53d406af60 msgr2=0x7f53d406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:28.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.525+0000 7f53f6fcb700 1 --2- 172.21.15.91:0/1916193200 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f53d406af60 0x7f53d406d420 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f53e40090b0 tx=0x7f53e401c040 comp rx=0 tx=0).stop 2023-11-24T06:09:28.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.525+0000 7f53f6fcb700 1 -- 172.21.15.91:0/1916193200 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53f01457b0 msgr2=0x7f53f015eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:28.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.525+0000 7f53f6fcb700 1 --2- 172.21.15.91:0/1916193200 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f53f01457b0 0x7f53f015eef0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f53e800cca0 tx=0x7f53e800a060 comp rx=0 tx=0).stop 2023-11-24T06:09:28.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.525+0000 7f53f5568700 1 -- 172.21.15.91:0/1916193200 reap_dead start 2023-11-24T06:09:28.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.525+0000 7f53f6fcb700 1 -- 172.21.15.91:0/1916193200 shutdown_connections 2023-11-24T06:09:28.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.525+0000 7f53f6fcb700 1 -- 172.21.15.91:0/1916193200 >> 172.21.15.91:0/1916193200 conn(0x7f53f00b2760 msgr2=0x7f53f0141340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:28.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.525+0000 7f53f6fcb700 1 -- 172.21.15.91:0/1916193200 shutdown_connections 2023-11-24T06:09:28.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.525+0000 7f53f6fcb700 1 -- 172.21.15.91:0/1916193200 wait complete. 2023-11-24T06:09:28.540 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-11-24T06:09:28.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.658+0000 7f469ef16700 1 Processor -- start 2023-11-24T06:09:28.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.658+0000 7f469ef16700 1 -- start start 2023-11-24T06:09:28.664 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.659+0000 7f469ef16700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4698158800 0x7f469815bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:28.664 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.659+0000 7f469ef16700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f46980b08a0 0x7f469806b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:28.665 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.659+0000 7f469ef16700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469815c130 con 0x7f46980b5f20 2023-11-24T06:09:28.665 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.659+0000 7f469ef16700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4698158be0 con 0x7f4698158800 2023-11-24T06:09:28.665 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.659+0000 7f469ef16700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46980726a0 con 0x7f46980b08a0 2023-11-24T06:09:28.665 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.659+0000 7f469ccb2700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f4698158800 0x7f469815bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:28.665 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.659+0000 7f4697fff700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f46980b5f20 0x7f46980b0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55336/0 (socket says 172.21.15.91:55336) 2023-11-24T06:09:28.665 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.659+0000 7f4697fff700 1 -- 172.21.15.91:0/86272837 learned_addr learned my addr 172.21.15.91:0/86272837 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:28.665 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.659+0000 7f469d4b3700 1 --2- 172.21.15.91:0/86272837 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f46980b08a0 0x7f469806b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:28.666 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.659+0000 7f469ccb2700 1 -- 172.21.15.91:0/86272837 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f46980b08a0 msgr2=0x7f469806b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:28.666 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.659+0000 7f469ccb2700 1 --2- 172.21.15.91:0/86272837 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f46980b08a0 0x7f469806b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:28.666 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.659+0000 7f469ccb2700 1 -- 172.21.15.91:0/86272837 >> v1:172.21.15.91:6789/0 conn(0x7f46980b5f20 legacy=0x7f46980b0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:28.666 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.659+0000 7f469ccb2700 1 -- 172.21.15.91:0/86272837 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46980728f0 con 0x7f4698158800 2023-11-24T06:09:28.666 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.659+0000 7f46977fe700 1 -- 172.21.15.91:0/86272837 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858161292 0 0) 0x7f4684001110 con 0x7f46980b5f20 2023-11-24T06:09:28.666 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.659+0000 7f469ccb2700 1 --2- 172.21.15.91:0/86272837 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4698158800 0x7f469815bbf0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f468c00bae0 tx=0x7f468c00bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cbb4078c020041b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:28.667 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.660+0000 7f46977fe700 1 -- 172.21.15.91:0/86272837 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c00dd30 con 0x7f4698158800 2023-11-24T06:09:28.667 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.660+0000 7f46977fe700 1 -- 172.21.15.91:0/86272837 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468c00de90 con 0x7f4698158800 2023-11-24T06:09:28.667 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.660+0000 7f46977fe700 1 -- 172.21.15.91:0/86272837 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c010690 con 0x7f4698158800 2023-11-24T06:09:28.667 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.660+0000 7f469ef16700 1 -- 172.21.15.91:0/86272837 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4698158800 msgr2=0x7f469815bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:28.667 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.660+0000 7f469ef16700 1 --2- 172.21.15.91:0/86272837 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4698158800 0x7f469815bbf0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f468c00bae0 tx=0x7f468c00bdf0 comp rx=0 tx=0).stop 2023-11-24T06:09:28.667 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.660+0000 7f469ef16700 1 -- 172.21.15.91:0/86272837 shutdown_connections 2023-11-24T06:09:28.667 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.660+0000 7f469ef16700 1 --2- 172.21.15.91:0/86272837 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f46980b08a0 0x7f469806b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:28.668 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.660+0000 7f469ef16700 1 --2- 172.21.15.91:0/86272837 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f4698158800 0x7f469815bbf0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:28.668 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.660+0000 7f469ef16700 1 -- 172.21.15.91:0/86272837 >> 172.21.15.91:0/86272837 conn(0x7f46980b50e0 msgr2=0x7f46980b57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:28.668 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.660+0000 7f469ef16700 1 -- 172.21.15.91:0/86272837 shutdown_connections 2023-11-24T06:09:28.668 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.660+0000 7f469ef16700 1 -- 172.21.15.91:0/86272837 wait complete. 2023-11-24T06:09:28.671 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.661+0000 7f469ef16700 1 Processor -- start 2023-11-24T06:09:28.671 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.661+0000 7f469ef16700 1 -- start start 2023-11-24T06:09:28.671 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.662+0000 7f469ef16700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f46980b08a0 0x7f4698150170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:28.671 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.662+0000 7f469ef16700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f46980b5f20 0x7f46980af340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:28.671 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.662+0000 7f469ef16700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4698158800 0x7f469806bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:28.672 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.662+0000 7f469ef16700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46980ae2a0 con 0x7f46980b5f20 2023-11-24T06:09:28.672 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.662+0000 7f469ef16700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46980ae580 con 0x7f4698158800 2023-11-24T06:09:28.672 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.662+0000 7f469ef16700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46980ae860 con 0x7f46980b08a0 2023-11-24T06:09:28.672 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.662+0000 7f4697fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f46980b5f20 0x7f46980af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:28.672 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.662+0000 7f4697fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f46980b5f20 0x7f46980af340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60408/0 (socket says 172.21.15.91:60408) 2023-11-24T06:09:28.672 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.662+0000 7f4697fff700 1 -- 172.21.15.91:0/2763497798 learned_addr learned my addr 172.21.15.91:0/2763497798 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:28.672 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.663+0000 7f469ccb2700 1 --2- 172.21.15.91:0/2763497798 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f46980b08a0 0x7f4698150170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:28.672 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.663+0000 7f469d4b3700 1 --2- 172.21.15.91:0/2763497798 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4698158800 0x7f469806bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:28.673 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.663+0000 7f4697fff700 1 -- 172.21.15.91:0/2763497798 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f46980b08a0 msgr2=0x7f4698150170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:28.673 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.663+0000 7f4697fff700 1 --2- 172.21.15.91:0/2763497798 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f46980b08a0 0x7f4698150170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:28.673 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.663+0000 7f4697fff700 1 -- 172.21.15.91:0/2763497798 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4698158800 msgr2=0x7f469806bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:28.673 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.663+0000 7f4697fff700 1 --2- 172.21.15.91:0/2763497798 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4698158800 0x7f469806bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:28.673 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.663+0000 7f4697fff700 1 -- 172.21.15.91:0/2763497798 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468c00b740 con 0x7f46980b5f20 2023-11-24T06:09:28.673 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.663+0000 7f469ccb2700 1 --2- 172.21.15.91:0/2763497798 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f46980b08a0 0x7f4698150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:28.673 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.663+0000 7f469d4b3700 1 --2- 172.21.15.91:0/2763497798 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4698158800 0x7f469806bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:28.674 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.663+0000 7f4697fff700 1 --2- 172.21.15.91:0/2763497798 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f46980b5f20 0x7f46980af340 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f468400c550 tx=0x7f468400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:28.674 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.663+0000 7f46957fa700 1 -- 172.21.15.91:0/2763497798 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684004c80 con 0x7f46980b5f20 2023-11-24T06:09:28.674 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.663+0000 7f46957fa700 1 -- 172.21.15.91:0/2763497798 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4684004e00 con 0x7f46980b5f20 2023-11-24T06:09:28.674 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.663+0000 7f469ef16700 1 -- 172.21.15.91:0/2763497798 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46980aeb40 con 0x7f46980b5f20 2023-11-24T06:09:28.674 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.663+0000 7f46957fa700 1 -- 172.21.15.91:0/2763497798 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684007310 con 0x7f46980b5f20 2023-11-24T06:09:28.674 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.663+0000 7f469ef16700 1 -- 172.21.15.91:0/2763497798 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46981506b0 con 0x7f46980b5f20 2023-11-24T06:09:28.674 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.664+0000 7f469ef16700 1 -- 172.21.15.91:0/2763497798 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f469806ca80 con 0x7f46980b5f20 2023-11-24T06:09:28.674 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.664+0000 7f46957fa700 1 -- 172.21.15.91:0/2763497798 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f4684007470 con 0x7f46980b5f20 2023-11-24T06:09:28.675 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.665+0000 7f46957fa700 1 --2- 172.21.15.91:0/2763497798 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f467c06af60 0x7f467c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:28.675 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.665+0000 7f46957fa700 1 -- 172.21.15.91:0/2763497798 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4684088e70 con 0x7f46980b5f20 2023-11-24T06:09:28.675 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.665+0000 7f469ccb2700 1 --2- 172.21.15.91:0/2763497798 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f467c06af60 0x7f467c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:28.675 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.665+0000 7f469ccb2700 1 --2- 172.21.15.91:0/2763497798 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f467c06af60 0x7f467c06d420 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f468c000c00 tx=0x7f468c013000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:28.675 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.669+0000 7f46957fa700 1 -- 172.21.15.91:0/2763497798 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46840542c0 con 0x7f46980b5f20 2023-11-24T06:09:28.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.861+0000 7f469ef16700 1 -- 172.21.15.91:0/2763497798 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f46980ae6f0 con 0x7f467c06af60 2023-11-24T06:09:28.865 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.862+0000 7f46957fa700 1 -- 172.21.15.91:0/2763497798 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f46980ae6f0 con 0x7f467c06af60 2023-11-24T06:09:28.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.864+0000 7f469ef16700 1 -- 172.21.15.91:0/2763497798 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f467c06af60 msgr2=0x7f467c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:28.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.864+0000 7f469ef16700 1 --2- 172.21.15.91:0/2763497798 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f467c06af60 0x7f467c06d420 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f468c000c00 tx=0x7f468c013000 comp rx=0 tx=0).stop 2023-11-24T06:09:28.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.864+0000 7f469ef16700 1 -- 172.21.15.91:0/2763497798 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f46980b5f20 msgr2=0x7f46980af340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:28.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.864+0000 7f469ef16700 1 --2- 172.21.15.91:0/2763497798 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f46980b5f20 0x7f46980af340 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f468400c550 tx=0x7f468400c910 comp rx=0 tx=0).stop 2023-11-24T06:09:28.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.864+0000 7f469ef16700 1 -- 172.21.15.91:0/2763497798 shutdown_connections 2023-11-24T06:09:28.868 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.864+0000 7f469ef16700 1 --2- 172.21.15.91:0/2763497798 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f467c06af60 0x7f467c06d420 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:28.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.864+0000 7f469ef16700 1 --2- 172.21.15.91:0/2763497798 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f4698158800 0x7f469806bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:28.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.864+0000 7f469ef16700 1 --2- 172.21.15.91:0/2763497798 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f46980b08a0 0x7f4698150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:28.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.864+0000 7f469ef16700 1 --2- 172.21.15.91:0/2763497798 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f46980b5f20 0x7f46980af340 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:28.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.864+0000 7f469ef16700 1 -- 172.21.15.91:0/2763497798 >> 172.21.15.91:0/2763497798 conn(0x7f46980b50e0 msgr2=0x7f46980b5580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:28.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.864+0000 7f469ef16700 1 -- 172.21.15.91:0/2763497798 shutdown_connections 2023-11-24T06:09:28.869 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:28.864+0000 7f469ef16700 1 -- 172.21.15.91:0/2763497798 wait complete. 2023-11-24T06:09:28.870 INFO:teuthology.orchestra.run.smithi091.stderr:instructed osd(s) 0 to deep-scrub 2023-11-24T06:09:28.880 INFO:tasks.ceph:Scrubbing osd.1 2023-11-24T06:09:28.880 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-11-24T06:09:29.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.000+0000 7f738bfdb700 1 Processor -- start 2023-11-24T06:09:29.005 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.001+0000 7f738bfdb700 1 -- start start 2023-11-24T06:09:29.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.001+0000 7f738bfdb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73840b59d0 0x7f738415e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.001+0000 7f738bfdb700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f7384072350 0x7f73840727c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.001+0000 7f738bfdb700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7384071ec0 con 0x7f738406afd0 2023-11-24T06:09:29.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.001+0000 7f738bfdb700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7384071b90 con 0x7f7384072350 2023-11-24T06:09:29.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.001+0000 7f738bfdb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738415ed30 con 0x7f73840b59d0 2023-11-24T06:09:29.009 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.002+0000 7f7389d77700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f738406afd0 0x7f738406b3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55348/0 (socket says 172.21.15.91:55348) 2023-11-24T06:09:29.009 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.002+0000 7f7389d77700 1 -- 172.21.15.91:0/2292945359 learned_addr learned my addr 172.21.15.91:0/2292945359 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:29.009 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.002+0000 7f7389576700 1 --2- 172.21.15.91:0/2292945359 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73840b59d0 0x7f738415e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.009 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.002+0000 7f738a578700 1 --2- 172.21.15.91:0/2292945359 >> v2:172.21.15.94:3300/0 conn(0x7f7384072350 0x7f73840727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.009 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.002+0000 7f7388d75700 1 -- 172.21.15.91:0/2292945359 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140932548 0 0) 0x7f7384071ec0 con 0x7f738406afd0 2023-11-24T06:09:29.009 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.002+0000 7f7388d75700 1 -- 172.21.15.91:0/2292945359 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73700035b0 con 0x7f738406afd0 2023-11-24T06:09:29.009 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.002+0000 7f7388d75700 1 -- 172.21.15.91:0/2292945359 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 334144356 0 0) 0x7f73700035b0 con 0x7f738406afd0 2023-11-24T06:09:29.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.002+0000 7f7389576700 1 -- 172.21.15.91:0/2292945359 >> v2:172.21.15.94:3300/0 conn(0x7f7384072350 msgr2=0x7f73840727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:29.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.002+0000 7f7389576700 1 --2- 172.21.15.91:0/2292945359 >> v2:172.21.15.94:3300/0 conn(0x7f7384072350 0x7f73840727c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.002+0000 7f7389576700 1 -- 172.21.15.91:0/2292945359 >> v1:172.21.15.91:6789/0 conn(0x7f738406afd0 legacy=0x7f738406b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.002+0000 7f7389576700 1 -- 172.21.15.91:0/2292945359 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738415f010 con 0x7f73840b59d0 2023-11-24T06:09:29.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.002+0000 7f738a578700 1 --2- 172.21.15.91:0/2292945359 >> v2:172.21.15.94:3300/0 conn(0x7f7384072350 0x7f73840727c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:09:29.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.002+0000 7f7389576700 1 --2- 172.21.15.91:0/2292945359 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73840b59d0 0x7f738415e470 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7380009840 tx=0x7f7380009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a98cec6a1fb94fbf server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:29.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.002+0000 7f7388d75700 1 -- 172.21.15.91:0/2292945359 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374001670 con 0x7f73840b59d0 2023-11-24T06:09:29.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.002+0000 7f7388d75700 1 -- 172.21.15.91:0/2292945359 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7380004030 con 0x7f73840b59d0 2023-11-24T06:09:29.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.003+0000 7f7388d75700 1 -- 172.21.15.91:0/2292945359 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73800043a0 con 0x7f73840b59d0 2023-11-24T06:09:29.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.003+0000 7f738bfdb700 1 -- 172.21.15.91:0/2292945359 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73840b59d0 msgr2=0x7f738415e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.003+0000 7f738bfdb700 1 --2- 172.21.15.91:0/2292945359 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73840b59d0 0x7f738415e470 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7380009840 tx=0x7f7380009b50 comp rx=0 tx=0).stop 2023-11-24T06:09:29.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.003+0000 7f738bfdb700 1 -- 172.21.15.91:0/2292945359 shutdown_connections 2023-11-24T06:09:29.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.003+0000 7f738bfdb700 1 --2- 172.21.15.91:0/2292945359 >> v2:172.21.15.94:3300/0 conn(0x7f7384072350 0x7f73840727c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.003+0000 7f738bfdb700 1 --2- 172.21.15.91:0/2292945359 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f73840b59d0 0x7f738415e470 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.003+0000 7f738bfdb700 1 -- 172.21.15.91:0/2292945359 >> 172.21.15.91:0/2292945359 conn(0x7f73840b2760 msgr2=0x7f73840b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:29.012 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.003+0000 7f738bfdb700 1 -- 172.21.15.91:0/2292945359 shutdown_connections 2023-11-24T06:09:29.012 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.003+0000 7f738bfdb700 1 -- 172.21.15.91:0/2292945359 wait complete. 2023-11-24T06:09:29.012 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.004+0000 7f738bfdb700 1 Processor -- start 2023-11-24T06:09:29.012 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.004+0000 7f738bfdb700 1 -- start start 2023-11-24T06:09:29.015 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f738bfdb700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f738406afd0 0x7f738415d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.015 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f738bfdb700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7384072350 0x7f73840c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.015 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f738bfdb700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f73840b59d0 0x7f7384146400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f738bfdb700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7384134950 con 0x7f73840b59d0 2023-11-24T06:09:29.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f738bfdb700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7384134c30 con 0x7f7384072350 2023-11-24T06:09:29.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f738bfdb700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7384134f10 con 0x7f738406afd0 2023-11-24T06:09:29.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f738a578700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f73840b59d0 0x7f7384146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f738a578700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f73840b59d0 0x7f7384146400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60424/0 (socket says 172.21.15.91:60424) 2023-11-24T06:09:29.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f738a578700 1 -- 172.21.15.91:0/3164454338 learned_addr learned my addr 172.21.15.91:0/3164454338 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:29.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f7389d77700 1 --2- 172.21.15.91:0/3164454338 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f738406afd0 0x7f738415d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f7389576700 1 --2- 172.21.15.91:0/3164454338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7384072350 0x7f73840c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f738a578700 1 -- 172.21.15.91:0/3164454338 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f738406afd0 msgr2=0x7f738415d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f738a578700 1 --2- 172.21.15.91:0/3164454338 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f738406afd0 0x7f738415d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f738a578700 1 -- 172.21.15.91:0/3164454338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7384072350 msgr2=0x7f73840c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f738a578700 1 --2- 172.21.15.91:0/3164454338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7384072350 0x7f73840c61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f738a578700 1 -- 172.21.15.91:0/3164454338 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737c009a70 con 0x7f73840b59d0 2023-11-24T06:09:29.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.005+0000 7f7389576700 1 --2- 172.21.15.91:0/3164454338 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f7384072350 0x7f73840c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:29.017 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.006+0000 7f738a578700 1 --2- 172.21.15.91:0/3164454338 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f73840b59d0 0x7f7384146400 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f737c00efc0 tx=0x7f737c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:29.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.006+0000 7f737a7fc700 1 -- 172.21.15.91:0/3164454338 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c009d70 con 0x7f73840b59d0 2023-11-24T06:09:29.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.006+0000 7f737a7fc700 1 -- 172.21.15.91:0/3164454338 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737c00f040 con 0x7f73840b59d0 2023-11-24T06:09:29.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.006+0000 7f738bfdb700 1 -- 172.21.15.91:0/3164454338 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73800094a0 con 0x7f73840b59d0 2023-11-24T06:09:29.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.006+0000 7f737a7fc700 1 -- 172.21.15.91:0/3164454338 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c00ce00 con 0x7f73840b59d0 2023-11-24T06:09:29.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.006+0000 7f738bfdb700 1 -- 172.21.15.91:0/3164454338 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73841354d0 con 0x7f73840b59d0 2023-11-24T06:09:29.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.006+0000 7f738bfdb700 1 -- 172.21.15.91:0/3164454338 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7384068ee0 con 0x7f73840b59d0 2023-11-24T06:09:29.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.007+0000 7f737a7fc700 1 -- 172.21.15.91:0/3164454338 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f737c014480 con 0x7f73840b59d0 2023-11-24T06:09:29.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.007+0000 7f737a7fc700 1 --2- 172.21.15.91:0/3164454338 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f736806af60 0x7f736806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.008+0000 7f7389d77700 1 --2- 172.21.15.91:0/3164454338 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f736806af60 0x7f736806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.008+0000 7f737a7fc700 1 -- 172.21.15.91:0/3164454338 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f737c089bf0 con 0x7f73840b59d0 2023-11-24T06:09:29.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.008+0000 7f737a7fc700 1 --2- 172.21.15.91:0/3164454338 >> [v2:172.21.15.91:6813/66406286,v1:172.21.15.91:6815/66406286] conn(0x7f7368070dc0 0x7f73680731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.008+0000 7f737a7fc700 1 -- 172.21.15.91:0/3164454338 --> [v2:172.21.15.91:6813/66406286,v1:172.21.15.91:6815/66406286] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73680738b0 con 0x7f7368070dc0 2023-11-24T06:09:29.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.008+0000 7f737a7fc700 1 -- 172.21.15.91:0/3164454338 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f737c08c300 con 0x7f73840b59d0 2023-11-24T06:09:29.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.008+0000 7f7389576700 1 --2- 172.21.15.91:0/3164454338 >> [v2:172.21.15.91:6813/66406286,v1:172.21.15.91:6815/66406286] conn(0x7f7368070dc0 0x7f73680731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.008+0000 7f7389d77700 1 --2- 172.21.15.91:0/3164454338 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f736806af60 0x7f736806d420 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7374001eb0 tx=0x7f7374001c60 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:29.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.008+0000 7f7389576700 1 --2- 172.21.15.91:0/3164454338 >> [v2:172.21.15.91:6813/66406286,v1:172.21.15.91:6815/66406286] conn(0x7f7368070dc0 0x7f73680731e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:29.020 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.009+0000 7f737a7fc700 1 -- 172.21.15.91:0/3164454338 <== osd.1 v2:172.21.15.91:6813/66406286 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f73680738b0 con 0x7f7368070dc0 2023-11-24T06:09:29.027 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.024+0000 7f738bfdb700 1 -- 172.21.15.91:0/3164454338 --> [v2:172.21.15.91:6813/66406286,v1:172.21.15.91:6815/66406286] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7384070050 con 0x7f7368070dc0 2023-11-24T06:09:29.028 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.024+0000 7f737a7fc700 1 -- 172.21.15.91:0/3164454338 <== osd.1 v2:172.21.15.91:6813/66406286 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f7384070050 con 0x7f7368070dc0 2023-11-24T06:09:29.028 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2023-11-24T06:09:29.028 INFO:teuthology.orchestra.run.smithi091.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T06:09:29.028 INFO:teuthology.orchestra.run.smithi091.stdout:} 2023-11-24T06:09:29.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.024+0000 7f738bfdb700 1 -- 172.21.15.91:0/3164454338 >> [v2:172.21.15.91:6813/66406286,v1:172.21.15.91:6815/66406286] conn(0x7f7368070dc0 msgr2=0x7f73680731e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.029 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.024+0000 7f738bfdb700 1 --2- 172.21.15.91:0/3164454338 >> [v2:172.21.15.91:6813/66406286,v1:172.21.15.91:6815/66406286] conn(0x7f7368070dc0 0x7f73680731e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.024+0000 7f738bfdb700 1 -- 172.21.15.91:0/3164454338 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f736806af60 msgr2=0x7f736806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.024+0000 7f738bfdb700 1 --2- 172.21.15.91:0/3164454338 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f736806af60 0x7f736806d420 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7374001eb0 tx=0x7f7374001c60 comp rx=0 tx=0).stop 2023-11-24T06:09:29.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.024+0000 7f738bfdb700 1 -- 172.21.15.91:0/3164454338 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f73840b59d0 msgr2=0x7f7384146400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.024+0000 7f738bfdb700 1 --2- 172.21.15.91:0/3164454338 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f73840b59d0 0x7f7384146400 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f737c00efc0 tx=0x7f737c00c3a0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.024+0000 7f738a578700 1 -- 172.21.15.91:0/3164454338 reap_dead start 2023-11-24T06:09:29.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.024+0000 7f738bfdb700 1 -- 172.21.15.91:0/3164454338 shutdown_connections 2023-11-24T06:09:29.030 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.024+0000 7f738bfdb700 1 -- 172.21.15.91:0/3164454338 >> 172.21.15.91:0/3164454338 conn(0x7f73840b2760 msgr2=0x7f738406b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:29.031 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.024+0000 7f738bfdb700 1 -- 172.21.15.91:0/3164454338 shutdown_connections 2023-11-24T06:09:29.031 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.024+0000 7f738bfdb700 1 -- 172.21.15.91:0/3164454338 wait complete. 2023-11-24T06:09:29.040 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-11-24T06:09:29.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.158+0000 7f84e92bc700 1 Processor -- start 2023-11-24T06:09:29.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.158+0000 7f84e92bc700 1 -- start start 2023-11-24T06:09:29.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.158+0000 7f84e92bc700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f84e415e060 0x7f84e40cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.158+0000 7f84e92bc700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f84e415e440 0x7f84e40ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.158+0000 7f84e92bc700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e40cd150 con 0x7f84e40d2750 2023-11-24T06:09:29.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.158+0000 7f84e92bc700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e40cfe80 con 0x7f84e415e060 2023-11-24T06:09:29.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.158+0000 7f84e92bc700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e40ccb00 con 0x7f84e415e440 2023-11-24T06:09:29.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.158+0000 7f84e2ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f84e415e060 0x7f84e40cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.158+0000 7f84e37fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f84e40d2750 0x7f84e406bf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55368/0 (socket says 172.21.15.91:55368) 2023-11-24T06:09:29.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.158+0000 7f84e37fe700 1 -- 172.21.15.91:0/3544213261 learned_addr learned my addr 172.21.15.91:0/3544213261 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:29.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.159+0000 7f84e27fc700 1 --2- 172.21.15.91:0/3544213261 >> v2:172.21.15.94:3300/0 conn(0x7f84e415e440 0x7f84e40ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.159+0000 7f84e2ffd700 1 -- 172.21.15.91:0/3544213261 >> v2:172.21.15.94:3300/0 conn(0x7f84e415e440 msgr2=0x7f84e40ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:29.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.159+0000 7f84e2ffd700 1 --2- 172.21.15.91:0/3544213261 >> v2:172.21.15.94:3300/0 conn(0x7f84e415e440 0x7f84e40ce5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.159+0000 7f84e2ffd700 1 -- 172.21.15.91:0/3544213261 >> v1:172.21.15.91:6789/0 conn(0x7f84e40d2750 legacy=0x7f84e406bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.159+0000 7f84e2ffd700 1 -- 172.21.15.91:0/3544213261 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e40ccde0 con 0x7f84e415e060 2023-11-24T06:09:29.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.159+0000 7f84e1ffb700 1 -- 172.21.15.91:0/3544213261 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126415673 0 0) 0x7f84e40cd150 con 0x7f84e40d2750 2023-11-24T06:09:29.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.159+0000 7f84e2ffd700 1 --2- 172.21.15.91:0/3544213261 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84e415e060 0x7f84e40cd5e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f84e406eb80 tx=0x7f84cc009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a2f9e114fa7b69b server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:29.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.159+0000 7f84e1ffb700 1 -- 172.21.15.91:0/3544213261 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc011830 con 0x7f84e415e060 2023-11-24T06:09:29.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.159+0000 7f84e1ffb700 1 -- 172.21.15.91:0/3544213261 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84cc011990 con 0x7f84e415e060 2023-11-24T06:09:29.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.159+0000 7f84e1ffb700 1 -- 172.21.15.91:0/3544213261 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc011d20 con 0x7f84e415e060 2023-11-24T06:09:29.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.160+0000 7f84e92bc700 1 -- 172.21.15.91:0/3544213261 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84e415e060 msgr2=0x7f84e40cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.160+0000 7f84e92bc700 1 --2- 172.21.15.91:0/3544213261 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84e415e060 0x7f84e40cd5e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f84e406eb80 tx=0x7f84cc009b20 comp rx=0 tx=0).stop 2023-11-24T06:09:29.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.160+0000 7f84e92bc700 1 -- 172.21.15.91:0/3544213261 shutdown_connections 2023-11-24T06:09:29.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.160+0000 7f84e92bc700 1 --2- 172.21.15.91:0/3544213261 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84e415e060 0x7f84e40cd5e0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.160+0000 7f84e92bc700 1 --2- 172.21.15.91:0/3544213261 >> v2:172.21.15.94:3300/0 conn(0x7f84e415e440 0x7f84e40ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.160+0000 7f84e92bc700 1 -- 172.21.15.91:0/3544213261 >> 172.21.15.91:0/3544213261 conn(0x7f84e40d1c30 msgr2=0x7f84e40d2040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:29.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.160+0000 7f84e92bc700 1 -- 172.21.15.91:0/3544213261 shutdown_connections 2023-11-24T06:09:29.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.160+0000 7f84e92bc700 1 -- 172.21.15.91:0/3544213261 wait complete. 2023-11-24T06:09:29.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.161+0000 7f84e92bc700 1 Processor -- start 2023-11-24T06:09:29.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.161+0000 7f84e92bc700 1 -- start start 2023-11-24T06:09:29.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e92bc700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84e40d2750 0x7f84e40b7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.171 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e92bc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84e415e060 0x7f84e40c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e92bc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84e415e440 0x7f84e41f7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e92bc700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e40cc1a0 con 0x7f84e415e060 2023-11-24T06:09:29.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e92bc700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e40cb620 con 0x7f84e40d2750 2023-11-24T06:09:29.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e92bc700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e40cb900 con 0x7f84e415e440 2023-11-24T06:09:29.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e27fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84e415e060 0x7f84e40c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e27fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84e415e060 0x7f84e40c8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60438/0 (socket says 172.21.15.91:60438) 2023-11-24T06:09:29.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e27fc700 1 -- 172.21.15.91:0/1538837685 learned_addr learned my addr 172.21.15.91:0/1538837685 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:29.172 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e2ffd700 1 --2- 172.21.15.91:0/1538837685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84e40d2750 0x7f84e40b7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e37fe700 1 --2- 172.21.15.91:0/1538837685 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84e415e440 0x7f84e41f7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e27fc700 1 -- 172.21.15.91:0/1538837685 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84e415e440 msgr2=0x7f84e41f7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e27fc700 1 --2- 172.21.15.91:0/1538837685 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84e415e440 0x7f84e41f7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e27fc700 1 -- 172.21.15.91:0/1538837685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84e40d2750 msgr2=0x7f84e40b7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e27fc700 1 --2- 172.21.15.91:0/1538837685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84e40d2750 0x7f84e40b7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e27fc700 1 -- 172.21.15.91:0/1538837685 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84cc0094a0 con 0x7f84e415e060 2023-11-24T06:09:29.173 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e2ffd700 1 --2- 172.21.15.91:0/1538837685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84e40d2750 0x7f84e40b7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:29.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e27fc700 1 --2- 172.21.15.91:0/1538837685 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84e415e060 0x7f84e40c8d00 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f84d400d800 tx=0x7f84d400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:29.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84db7fe700 1 -- 172.21.15.91:0/1538837685 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d40125f0 con 0x7f84e415e060 2023-11-24T06:09:29.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84db7fe700 1 -- 172.21.15.91:0/1538837685 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d4012750 con 0x7f84e415e060 2023-11-24T06:09:29.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.162+0000 7f84e92bc700 1 -- 172.21.15.91:0/1538837685 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e40cbbe0 con 0x7f84e415e060 2023-11-24T06:09:29.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.163+0000 7f84db7fe700 1 -- 172.21.15.91:0/1538837685 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4012b00 con 0x7f84e415e060 2023-11-24T06:09:29.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.163+0000 7f84e92bc700 1 -- 172.21.15.91:0/1538837685 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e414df50 con 0x7f84e415e060 2023-11-24T06:09:29.174 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.163+0000 7f84e92bc700 1 -- 172.21.15.91:0/1538837685 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e4067f60 con 0x7f84e415e060 2023-11-24T06:09:29.175 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.164+0000 7f84db7fe700 1 -- 172.21.15.91:0/1538837685 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f84d4012c60 con 0x7f84e415e060 2023-11-24T06:09:29.175 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.164+0000 7f84db7fe700 1 --2- 172.21.15.91:0/1538837685 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f84c406af60 0x7f84c406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.175 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.164+0000 7f84e2ffd700 1 --2- 172.21.15.91:0/1538837685 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f84c406af60 0x7f84c406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.175 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.164+0000 7f84db7fe700 1 -- 172.21.15.91:0/1538837685 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f84d4089670 con 0x7f84e415e060 2023-11-24T06:09:29.175 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.165+0000 7f84e2ffd700 1 --2- 172.21.15.91:0/1538837685 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f84c406af60 0x7f84c406d420 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f84cc0090b0 tx=0x7f84cc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:29.176 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.169+0000 7f84db7fe700 1 -- 172.21.15.91:0/1538837685 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f84d4054ac0 con 0x7f84e415e060 2023-11-24T06:09:29.363 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.358+0000 7f84e92bc700 1 -- 172.21.15.91:0/1538837685 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f84e406d8d0 con 0x7f84c406af60 2023-11-24T06:09:29.363 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.358+0000 7f84db7fe700 1 -- 172.21.15.91:0/1538837685 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f84e406d8d0 con 0x7f84c406af60 2023-11-24T06:09:29.366 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.360+0000 7f84e92bc700 1 -- 172.21.15.91:0/1538837685 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f84c406af60 msgr2=0x7f84c406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.366 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.360+0000 7f84e92bc700 1 --2- 172.21.15.91:0/1538837685 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f84c406af60 0x7f84c406d420 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f84cc0090b0 tx=0x7f84cc008030 comp rx=0 tx=0).stop 2023-11-24T06:09:29.366 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.360+0000 7f84e92bc700 1 -- 172.21.15.91:0/1538837685 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84e415e060 msgr2=0x7f84e40c8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.367 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.360+0000 7f84e92bc700 1 --2- 172.21.15.91:0/1538837685 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84e415e060 0x7f84e40c8d00 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f84d400d800 tx=0x7f84d400dbc0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.367 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.360+0000 7f84e92bc700 1 -- 172.21.15.91:0/1538837685 shutdown_connections 2023-11-24T06:09:29.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.360+0000 7f84e92bc700 1 --2- 172.21.15.91:0/1538837685 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f84c406af60 0x7f84c406d420 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.368 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.360+0000 7f84e92bc700 1 --2- 172.21.15.91:0/1538837685 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f84e40d2750 0x7f84e40b7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.369 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.360+0000 7f84e92bc700 1 --2- 172.21.15.91:0/1538837685 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f84e415e440 0x7f84e41f7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.369 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.360+0000 7f84e92bc700 1 --2- 172.21.15.91:0/1538837685 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f84e415e060 0x7f84e40c8d00 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.369 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.360+0000 7f84e92bc700 1 -- 172.21.15.91:0/1538837685 >> 172.21.15.91:0/1538837685 conn(0x7f84e40d1c30 msgr2=0x7f84e40d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:29.370 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.360+0000 7f84e92bc700 1 -- 172.21.15.91:0/1538837685 shutdown_connections 2023-11-24T06:09:29.370 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.360+0000 7f84e92bc700 1 -- 172.21.15.91:0/1538837685 wait complete. 2023-11-24T06:09:29.370 INFO:teuthology.orchestra.run.smithi091.stderr:instructed osd(s) 1 to deep-scrub 2023-11-24T06:09:29.377 INFO:tasks.ceph:Scrubbing osd.2 2023-11-24T06:09:29.378 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-11-24T06:09:29.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.500+0000 7febce3d6700 1 Processor -- start 2023-11-24T06:09:29.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.500+0000 7febce3d6700 1 -- start start 2023-11-24T06:09:29.506 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.500+0000 7febce3d6700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7febc8070650 0x7febc80b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.506 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.500+0000 7febce3d6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7febc80afcd0 0x7febc80b54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.506 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.500+0000 7febce3d6700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc806ee80 con 0x7febc80af1e0 2023-11-24T06:09:29.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febce3d6700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febc806f220 con 0x7febc8070650 2023-11-24T06:09:29.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febce3d6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc80b5b30 con 0x7febc80afcd0 2023-11-24T06:09:29.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febc77fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7febc8070650 0x7febc80b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febc7fff700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7febc80af1e0 0x7febc80af5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55378/0 (socket says 172.21.15.91:55378) 2023-11-24T06:09:29.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febc77fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7febc8070650 0x7febc80b08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60446/0 (socket says 172.21.15.91:60446) 2023-11-24T06:09:29.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febc7fff700 1 -- 172.21.15.91:0/1819099486 learned_addr learned my addr 172.21.15.91:0/1819099486 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:29.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febcc973700 1 --2- 172.21.15.91:0/1819099486 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7febc80afcd0 0x7febc80b54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febc77fe700 1 -- 172.21.15.91:0/1819099486 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7febc80afcd0 msgr2=0x7febc80b54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:29.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febc77fe700 1 --2- 172.21.15.91:0/1819099486 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7febc80afcd0 0x7febc80b54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febc6ffd700 1 -- 172.21.15.91:0/1819099486 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514851887 0 0) 0x7febc806ee80 con 0x7febc80af1e0 2023-11-24T06:09:29.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febc77fe700 1 -- 172.21.15.91:0/1819099486 >> v1:172.21.15.91:6789/0 conn(0x7febc80af1e0 legacy=0x7febc80af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febc77fe700 1 -- 172.21.15.91:0/1819099486 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc80b5e10 con 0x7febc8070650 2023-11-24T06:09:29.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febcc973700 1 --2- 172.21.15.91:0/1819099486 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7febc80afcd0 0x7febc80b54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:29.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febc77fe700 1 --2- 172.21.15.91:0/1819099486 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7febc8070650 0x7febc80b08c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7febbc009840 tx=0x7febbc009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86283edb5fce9175 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:29.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febc6ffd700 1 -- 172.21.15.91:0/1819099486 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc011830 con 0x7febc8070650 2023-11-24T06:09:29.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febc6ffd700 1 -- 172.21.15.91:0/1819099486 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febbc011990 con 0x7febc8070650 2023-11-24T06:09:29.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.501+0000 7febc6ffd700 1 -- 172.21.15.91:0/1819099486 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc011d20 con 0x7febc8070650 2023-11-24T06:09:29.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.502+0000 7febce3d6700 1 -- 172.21.15.91:0/1819099486 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7febc8070650 msgr2=0x7febc80b08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.502+0000 7febce3d6700 1 --2- 172.21.15.91:0/1819099486 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7febc8070650 0x7febc80b08c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7febbc009840 tx=0x7febbc009b50 comp rx=0 tx=0).stop 2023-11-24T06:09:29.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.502+0000 7febce3d6700 1 -- 172.21.15.91:0/1819099486 shutdown_connections 2023-11-24T06:09:29.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.502+0000 7febce3d6700 1 --2- 172.21.15.91:0/1819099486 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7febc80afcd0 0x7febc80b54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.502+0000 7febce3d6700 1 --2- 172.21.15.91:0/1819099486 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7febc8070650 0x7febc80b08c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.502+0000 7febce3d6700 1 -- 172.21.15.91:0/1819099486 >> 172.21.15.91:0/1819099486 conn(0x7febc809a9f0 msgr2=0x7febc809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:29.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.502+0000 7febce3d6700 1 -- 172.21.15.91:0/1819099486 shutdown_connections 2023-11-24T06:09:29.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.502+0000 7febce3d6700 1 -- 172.21.15.91:0/1819099486 wait complete. 2023-11-24T06:09:29.513 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.503+0000 7febce3d6700 1 Processor -- start 2023-11-24T06:09:29.513 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.503+0000 7febce3d6700 1 -- start start 2023-11-24T06:09:29.513 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.504+0000 7febce3d6700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7febc8070650 0x7febc814c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.513 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.504+0000 7febce3d6700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7febc80af1e0 0x7febc80b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.504+0000 7febce3d6700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7febc80afcd0 0x7febc80b51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.504+0000 7febce3d6700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc8151e80 con 0x7febc80afcd0 2023-11-24T06:09:29.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.504+0000 7febce3d6700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc814c790 con 0x7febc80af1e0 2023-11-24T06:09:29.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.504+0000 7febce3d6700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc814fd30 con 0x7febc8070650 2023-11-24T06:09:29.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.504+0000 7febcc973700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7febc80afcd0 0x7febc80b51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.504+0000 7febcc973700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7febc80afcd0 0x7febc80b51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60452/0 (socket says 172.21.15.91:60452) 2023-11-24T06:09:29.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.504+0000 7febcc973700 1 -- 172.21.15.91:0/4106398465 learned_addr learned my addr 172.21.15.91:0/4106398465 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:29.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.504+0000 7febc7fff700 1 --2- 172.21.15.91:0/4106398465 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7febc8070650 0x7febc814c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.504+0000 7febc77fe700 1 --2- 172.21.15.91:0/4106398465 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7febc80af1e0 0x7febc80b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.504+0000 7febcc973700 1 -- 172.21.15.91:0/4106398465 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7febc8070650 msgr2=0x7febc814c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.504+0000 7febcc973700 1 --2- 172.21.15.91:0/4106398465 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7febc8070650 0x7febc814c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.504+0000 7febcc973700 1 -- 172.21.15.91:0/4106398465 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7febc80af1e0 msgr2=0x7febc80b4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.504+0000 7febcc973700 1 --2- 172.21.15.91:0/4106398465 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7febc80af1e0 0x7febc80b4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.504+0000 7febcc973700 1 -- 172.21.15.91:0/4106398465 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febbc0094a0 con 0x7febc80afcd0 2023-11-24T06:09:29.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.505+0000 7febcc973700 1 --2- 172.21.15.91:0/4106398465 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7febc80afcd0 0x7febc80b51e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7febc0007ef0 tx=0x7febc000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:29.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.505+0000 7febc4ff9700 1 -- 172.21.15.91:0/4106398465 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc000de60 con 0x7febc80afcd0 2023-11-24T06:09:29.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.505+0000 7febc4ff9700 1 -- 172.21.15.91:0/4106398465 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc000f040 con 0x7febc80afcd0 2023-11-24T06:09:29.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.505+0000 7febce3d6700 1 -- 172.21.15.91:0/4106398465 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc8150010 con 0x7febc80afcd0 2023-11-24T06:09:29.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.505+0000 7febc4ff9700 1 -- 172.21.15.91:0/4106398465 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc0003c60 con 0x7febc80afcd0 2023-11-24T06:09:29.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.505+0000 7febce3d6700 1 -- 172.21.15.91:0/4106398465 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc81505d0 con 0x7febc80afcd0 2023-11-24T06:09:29.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.505+0000 7febce3d6700 1 -- 172.21.15.91:0/4106398465 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7febc8068670 con 0x7febc80afcd0 2023-11-24T06:09:29.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.506+0000 7febc4ff9700 1 -- 172.21.15.91:0/4106398465 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7febc0004030 con 0x7febc80afcd0 2023-11-24T06:09:29.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.506+0000 7febc4ff9700 1 --2- 172.21.15.91:0/4106398465 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7febb406af60 0x7febb406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.506+0000 7febc4ff9700 1 -- 172.21.15.91:0/4106398465 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7febc0089610 con 0x7febc80afcd0 2023-11-24T06:09:29.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.506+0000 7febc7fff700 1 --2- 172.21.15.91:0/4106398465 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7febb406af60 0x7febb406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.507+0000 7febc4ff9700 1 --2- 172.21.15.91:0/4106398465 >> [v2:172.21.15.91:6808/649116497,v1:172.21.15.91:6809/649116497] conn(0x7febb4070dc0 0x7febb40731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.507+0000 7febc4ff9700 1 -- 172.21.15.91:0/4106398465 --> [v2:172.21.15.91:6808/649116497,v1:172.21.15.91:6809/649116497] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7febb40738b0 con 0x7febb4070dc0 2023-11-24T06:09:29.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.507+0000 7febc4ff9700 1 -- 172.21.15.91:0/4106398465 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7febc008bd20 con 0x7febc80afcd0 2023-11-24T06:09:29.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.507+0000 7febc77fe700 1 --2- 172.21.15.91:0/4106398465 >> [v2:172.21.15.91:6808/649116497,v1:172.21.15.91:6809/649116497] conn(0x7febb4070dc0 0x7febb40731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.507+0000 7febc7fff700 1 --2- 172.21.15.91:0/4106398465 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7febb406af60 0x7febb406d420 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7febb8000f60 tx=0x7febb8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:29.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.507+0000 7febc77fe700 1 --2- 172.21.15.91:0/4106398465 >> [v2:172.21.15.91:6808/649116497,v1:172.21.15.91:6809/649116497] conn(0x7febb4070dc0 0x7febb40731e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:29.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.508+0000 7febc4ff9700 1 -- 172.21.15.91:0/4106398465 <== osd.2 v2:172.21.15.91:6808/649116497 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7febb40738b0 con 0x7febb4070dc0 2023-11-24T06:09:29.526 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.522+0000 7febce3d6700 1 -- 172.21.15.91:0/4106398465 --> [v2:172.21.15.91:6808/649116497,v1:172.21.15.91:6809/649116497] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7febc81507f0 con 0x7febb4070dc0 2023-11-24T06:09:29.526 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.523+0000 7febc4ff9700 1 -- 172.21.15.91:0/4106398465 <== osd.2 v2:172.21.15.91:6808/649116497 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7febc81507f0 con 0x7febb4070dc0 2023-11-24T06:09:29.527 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2023-11-24T06:09:29.527 INFO:teuthology.orchestra.run.smithi091.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T06:09:29.527 INFO:teuthology.orchestra.run.smithi091.stdout:} 2023-11-24T06:09:29.529 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.523+0000 7febce3d6700 1 -- 172.21.15.91:0/4106398465 >> [v2:172.21.15.91:6808/649116497,v1:172.21.15.91:6809/649116497] conn(0x7febb4070dc0 msgr2=0x7febb40731e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.529 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.523+0000 7febce3d6700 1 --2- 172.21.15.91:0/4106398465 >> [v2:172.21.15.91:6808/649116497,v1:172.21.15.91:6809/649116497] conn(0x7febb4070dc0 0x7febb40731e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.529 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.523+0000 7febce3d6700 1 -- 172.21.15.91:0/4106398465 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7febb406af60 msgr2=0x7febb406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.529 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.523+0000 7febce3d6700 1 --2- 172.21.15.91:0/4106398465 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7febb406af60 0x7febb406d420 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7febb8000f60 tx=0x7febb8009040 comp rx=0 tx=0).stop 2023-11-24T06:09:29.529 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.523+0000 7febce3d6700 1 -- 172.21.15.91:0/4106398465 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7febc80afcd0 msgr2=0x7febc80b51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.523+0000 7febce3d6700 1 --2- 172.21.15.91:0/4106398465 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7febc80afcd0 0x7febc80b51e0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7febc0007ef0 tx=0x7febc000d3b0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.523+0000 7febcc973700 1 -- 172.21.15.91:0/4106398465 reap_dead start 2023-11-24T06:09:29.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.523+0000 7febce3d6700 1 -- 172.21.15.91:0/4106398465 shutdown_connections 2023-11-24T06:09:29.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.523+0000 7febce3d6700 1 -- 172.21.15.91:0/4106398465 >> 172.21.15.91:0/4106398465 conn(0x7febc809a9f0 msgr2=0x7febc809f330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:29.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.523+0000 7febce3d6700 1 -- 172.21.15.91:0/4106398465 shutdown_connections 2023-11-24T06:09:29.530 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.523+0000 7febce3d6700 1 -- 172.21.15.91:0/4106398465 wait complete. 2023-11-24T06:09:29.538 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-11-24T06:09:29.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.655+0000 7ff9713b4700 1 Processor -- start 2023-11-24T06:09:29.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.655+0000 7ff9713b4700 1 -- start start 2023-11-24T06:09:29.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.656+0000 7ff9713b4700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff96c1478c0 0x7ff96c0b57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.656+0000 7ff9713b4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff96c161a90 0x7ff96c06afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.656+0000 7ff9713b4700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff96c06dbe0 con 0x7ff96c06b510 2023-11-24T06:09:29.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.656+0000 7ff9713b4700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff96c06b9a0 con 0x7ff96c1478c0 2023-11-24T06:09:29.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.656+0000 7ff9713b4700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c147ca0 con 0x7ff96c161a90 2023-11-24T06:09:29.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.656+0000 7ff96a7fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff96c161a90 0x7ff96c06afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.656+0000 7ff96affd700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7ff96c1478c0 0x7ff96c0b57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.657+0000 7ff96a7fc700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff96c161a90 0x7ff96c06afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:43892/0 (socket says 172.21.15.91:43892) 2023-11-24T06:09:29.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.657+0000 7ff96a7fc700 1 -- 172.21.15.91:0/1538362165 learned_addr learned my addr 172.21.15.91:0/1538362165 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:29.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.657+0000 7ff96a7fc700 1 -- 172.21.15.91:0/1538362165 >> v2:172.21.15.94:3300/0 conn(0x7ff96c1478c0 msgr2=0x7ff96c0b57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:29.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.657+0000 7ff96a7fc700 1 --2- 172.21.15.91:0/1538362165 >> v2:172.21.15.94:3300/0 conn(0x7ff96c1478c0 0x7ff96c0b57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.657+0000 7ff96a7fc700 1 -- 172.21.15.91:0/1538362165 >> v1:172.21.15.94:6789/0 conn(0x7ff96c06b510 legacy=0x7ff96c071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.657+0000 7ff969ffb700 1 -- 172.21.15.91:0/1538362165 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189721477 0 0) 0x7ff96c06dbe0 con 0x7ff96c06b510 2023-11-24T06:09:29.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.657+0000 7ff96a7fc700 1 -- 172.21.15.91:0/1538362165 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff96c1622e0 con 0x7ff96c161a90 2023-11-24T06:09:29.664 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.657+0000 7ff96a7fc700 1 --2- 172.21.15.91:0/1538362165 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff96c161a90 0x7ff96c06afd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff95c0098f0 tx=0x7ff95c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5a1aa0b3ac2d7eb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:29.664 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.657+0000 7ff969ffb700 1 -- 172.21.15.91:0/1538362165 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c011830 con 0x7ff96c161a90 2023-11-24T06:09:29.664 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.657+0000 7ff969ffb700 1 -- 172.21.15.91:0/1538362165 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95c011990 con 0x7ff96c161a90 2023-11-24T06:09:29.664 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.657+0000 7ff969ffb700 1 -- 172.21.15.91:0/1538362165 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c011d20 con 0x7ff96c161a90 2023-11-24T06:09:29.664 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.658+0000 7ff9713b4700 1 -- 172.21.15.91:0/1538362165 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff96c161a90 msgr2=0x7ff96c06afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.664 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.658+0000 7ff9713b4700 1 --2- 172.21.15.91:0/1538362165 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff96c161a90 0x7ff96c06afd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff95c0098f0 tx=0x7ff95c009c00 comp rx=0 tx=0).stop 2023-11-24T06:09:29.664 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.658+0000 7ff9713b4700 1 -- 172.21.15.91:0/1538362165 shutdown_connections 2023-11-24T06:09:29.664 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.658+0000 7ff9713b4700 1 --2- 172.21.15.91:0/1538362165 >> v2:172.21.15.94:3300/0 conn(0x7ff96c1478c0 0x7ff96c0b57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.664 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.658+0000 7ff9713b4700 1 --2- 172.21.15.91:0/1538362165 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff96c161a90 0x7ff96c06afd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.665 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.658+0000 7ff9713b4700 1 -- 172.21.15.91:0/1538362165 >> 172.21.15.91:0/1538362165 conn(0x7ff96c0d0aa0 msgr2=0x7ff96c0d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:29.665 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.658+0000 7ff9713b4700 1 -- 172.21.15.91:0/1538362165 shutdown_connections 2023-11-24T06:09:29.665 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.658+0000 7ff9713b4700 1 -- 172.21.15.91:0/1538362165 wait complete. 2023-11-24T06:09:29.668 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.659+0000 7ff9713b4700 1 Processor -- start 2023-11-24T06:09:29.668 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.659+0000 7ff9713b4700 1 -- start start 2023-11-24T06:09:29.668 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.660+0000 7ff9713b4700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff96c06b510 0x7ff96c1e90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.668 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.660+0000 7ff9713b4700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff96c1478c0 0x7ff96c15f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.668 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.660+0000 7ff9713b4700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff96c161a90 0x7ff96c0b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.669 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.660+0000 7ff9713b4700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c1e9b30 con 0x7ff96c1478c0 2023-11-24T06:09:29.669 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.660+0000 7ff9713b4700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c1e9e10 con 0x7ff96c161a90 2023-11-24T06:09:29.669 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.660+0000 7ff9713b4700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c0b10b0 con 0x7ff96c06b510 2023-11-24T06:09:29.669 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.660+0000 7ff96a7fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff96c1478c0 0x7ff96c15f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.669 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.660+0000 7ff96a7fc700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff96c1478c0 0x7ff96c15f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60468/0 (socket says 172.21.15.91:60468) 2023-11-24T06:09:29.669 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.660+0000 7ff96a7fc700 1 -- 172.21.15.91:0/1774386770 learned_addr learned my addr 172.21.15.91:0/1774386770 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:29.669 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.660+0000 7ff96affd700 1 --2- 172.21.15.91:0/1774386770 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff96c06b510 0x7ff96c1e90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.669 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.660+0000 7ff96b7fe700 1 --2- 172.21.15.91:0/1774386770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff96c161a90 0x7ff96c0b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.670 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.660+0000 7ff96a7fc700 1 -- 172.21.15.91:0/1774386770 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff96c06b510 msgr2=0x7ff96c1e90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.670 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.660+0000 7ff96a7fc700 1 --2- 172.21.15.91:0/1774386770 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff96c06b510 0x7ff96c1e90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.670 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.660+0000 7ff96a7fc700 1 -- 172.21.15.91:0/1774386770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff96c161a90 msgr2=0x7ff96c0b18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.670 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.660+0000 7ff96a7fc700 1 --2- 172.21.15.91:0/1774386770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff96c161a90 0x7ff96c0b18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.670 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.660+0000 7ff96a7fc700 1 -- 172.21.15.91:0/1774386770 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95c009550 con 0x7ff96c1478c0 2023-11-24T06:09:29.670 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.661+0000 7ff96a7fc700 1 --2- 172.21.15.91:0/1774386770 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff96c1478c0 0x7ff96c15f8c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff95c014040 tx=0x7ff95c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:29.670 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.661+0000 7ff9537fe700 1 -- 172.21.15.91:0/1774386770 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c0036a0 con 0x7ff96c1478c0 2023-11-24T06:09:29.670 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.661+0000 7ff9537fe700 1 -- 172.21.15.91:0/1774386770 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95c01f070 con 0x7ff96c1478c0 2023-11-24T06:09:29.671 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.661+0000 7ff9713b4700 1 -- 172.21.15.91:0/1774386770 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff96c0b1390 con 0x7ff96c1478c0 2023-11-24T06:09:29.671 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.661+0000 7ff9537fe700 1 -- 172.21.15.91:0/1774386770 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c003a40 con 0x7ff96c1478c0 2023-11-24T06:09:29.671 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.661+0000 7ff9713b4700 1 -- 172.21.15.91:0/1774386770 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff96c15fe00 con 0x7ff96c1478c0 2023-11-24T06:09:29.671 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.662+0000 7ff9713b4700 1 -- 172.21.15.91:0/1774386770 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff96c068670 con 0x7ff96c1478c0 2023-11-24T06:09:29.671 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.662+0000 7ff9537fe700 1 -- 172.21.15.91:0/1774386770 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7ff95c003ba0 con 0x7ff96c1478c0 2023-11-24T06:09:29.671 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.662+0000 7ff9537fe700 1 --2- 172.21.15.91:0/1774386770 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff95806af60 0x7ff95806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:29.672 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.663+0000 7ff9537fe700 1 -- 172.21.15.91:0/1774386770 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff95c021070 con 0x7ff96c1478c0 2023-11-24T06:09:29.672 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.663+0000 7ff96affd700 1 --2- 172.21.15.91:0/1774386770 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff95806af60 0x7ff95806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:29.672 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.663+0000 7ff96affd700 1 --2- 172.21.15.91:0/1774386770 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff95806af60 0x7ff95806d420 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff96c0cfc50 tx=0x7ff960009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:29.672 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.667+0000 7ff9537fe700 1 -- 172.21.15.91:0/1774386770 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff95c055fa0 con 0x7ff96c1478c0 2023-11-24T06:09:29.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.856+0000 7ff9713b4700 1 -- 172.21.15.91:0/1774386770 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff96c1628a0 con 0x7ff95806af60 2023-11-24T06:09:29.861 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.857+0000 7ff9537fe700 1 -- 172.21.15.91:0/1774386770 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff960011910 con 0x7ff95806af60 2023-11-24T06:09:29.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.858+0000 7ff9713b4700 1 -- 172.21.15.91:0/1774386770 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff95806af60 msgr2=0x7ff95806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.859+0000 7ff9713b4700 1 --2- 172.21.15.91:0/1774386770 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff95806af60 0x7ff95806d420 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff96c0cfc50 tx=0x7ff960009210 comp rx=0 tx=0).stop 2023-11-24T06:09:29.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.859+0000 7ff9713b4700 1 -- 172.21.15.91:0/1774386770 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff96c1478c0 msgr2=0x7ff96c15f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:29.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.859+0000 7ff9713b4700 1 --2- 172.21.15.91:0/1774386770 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff96c1478c0 0x7ff96c15f8c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff95c014040 tx=0x7ff95c004750 comp rx=0 tx=0).stop 2023-11-24T06:09:29.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.859+0000 7ff9713b4700 1 -- 172.21.15.91:0/1774386770 shutdown_connections 2023-11-24T06:09:29.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.859+0000 7ff9713b4700 1 --2- 172.21.15.91:0/1774386770 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff95806af60 0x7ff95806d420 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.859+0000 7ff9713b4700 1 --2- 172.21.15.91:0/1774386770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff96c161a90 0x7ff96c0b18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.859+0000 7ff9713b4700 1 --2- 172.21.15.91:0/1774386770 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff96c06b510 0x7ff96c1e90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.859+0000 7ff9713b4700 1 --2- 172.21.15.91:0/1774386770 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff96c1478c0 0x7ff96c15f8c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:29.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.859+0000 7ff9713b4700 1 -- 172.21.15.91:0/1774386770 >> 172.21.15.91:0/1774386770 conn(0x7ff96c0d0aa0 msgr2=0x7ff96c1451e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:29.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.859+0000 7ff9713b4700 1 -- 172.21.15.91:0/1774386770 shutdown_connections 2023-11-24T06:09:29.864 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.859+0000 7ff9713b4700 1 -- 172.21.15.91:0/1774386770 wait complete. 2023-11-24T06:09:29.865 INFO:teuthology.orchestra.run.smithi091.stderr:instructed osd(s) 2 to deep-scrub 2023-11-24T06:09:29.874 INFO:tasks.ceph:Scrubbing osd.3 2023-11-24T06:09:29.875 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-11-24T06:09:29.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.995+0000 7faff6bf0700 1 Processor -- start 2023-11-24T06:09:29.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.995+0000 7faff6bf0700 1 -- start start 2023-11-24T06:09:30.000 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.996+0000 7faff6bf0700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faff006afd0 0x7faff00b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.996+0000 7faff6bf0700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7faff006b3b0 0x7faff015e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.996+0000 7faff6bf0700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff0071ec0 con 0x7faff0072350 2023-11-24T06:09:30.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.996+0000 7faff6bf0700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff0071b90 con 0x7faff006afd0 2023-11-24T06:09:30.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.996+0000 7faff6bf0700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff015ed30 con 0x7faff006b3b0 2023-11-24T06:09:30.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.996+0000 7fafeffff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7faff006b3b0 0x7faff015e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.996+0000 7faff498c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faff006afd0 0x7faff00b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.996+0000 7fafeffff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7faff006b3b0 0x7faff015e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:43908/0 (socket says 172.21.15.91:43908) 2023-11-24T06:09:30.001 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.996+0000 7faff498c700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7faff006afd0 0x7faff00b59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:44866/0 (socket says 172.21.15.91:44866) 2023-11-24T06:09:30.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.996+0000 7fafeffff700 1 -- 172.21.15.91:0/654423051 learned_addr learned my addr 172.21.15.91:0/654423051 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:30.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.996+0000 7faff498c700 1 -- 172.21.15.91:0/654423051 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7faff006b3b0 msgr2=0x7faff015e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:30.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.996+0000 7faff498c700 1 --2- 172.21.15.91:0/654423051 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7faff006b3b0 0x7faff015e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.996+0000 7faff498c700 1 -- 172.21.15.91:0/654423051 >> v1:172.21.15.94:6789/0 conn(0x7faff0072350 legacy=0x7faff006dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.996+0000 7faff498c700 1 -- 172.21.15.91:0/654423051 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff015f010 con 0x7faff006afd0 2023-11-24T06:09:30.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.996+0000 7faff498c700 1 --2- 172.21.15.91:0/654423051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff006afd0 0x7faff00b59d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fafe4009840 tx=0x7fafe4009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83e94b1002db357a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:30.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.997+0000 7fafef7fe700 1 -- 172.21.15.91:0/654423051 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe4011830 con 0x7faff006afd0 2023-11-24T06:09:30.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.997+0000 7fafef7fe700 1 -- 172.21.15.91:0/654423051 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe4011990 con 0x7faff006afd0 2023-11-24T06:09:30.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.997+0000 7fafef7fe700 1 -- 172.21.15.91:0/654423051 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe4011d20 con 0x7faff006afd0 2023-11-24T06:09:30.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.997+0000 7faff6bf0700 1 -- 172.21.15.91:0/654423051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff006afd0 msgr2=0x7faff00b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.997+0000 7faff6bf0700 1 --2- 172.21.15.91:0/654423051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff006afd0 0x7faff00b59d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fafe4009840 tx=0x7fafe4009b50 comp rx=0 tx=0).stop 2023-11-24T06:09:30.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.997+0000 7faff6bf0700 1 -- 172.21.15.91:0/654423051 shutdown_connections 2023-11-24T06:09:30.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.997+0000 7faff6bf0700 1 --2- 172.21.15.91:0/654423051 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff006afd0 0x7faff00b59d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.997+0000 7faff6bf0700 1 --2- 172.21.15.91:0/654423051 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7faff006b3b0 0x7faff015e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.997+0000 7faff6bf0700 1 -- 172.21.15.91:0/654423051 >> 172.21.15.91:0/654423051 conn(0x7faff00b2760 msgr2=0x7faff00b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:30.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.997+0000 7faff6bf0700 1 -- 172.21.15.91:0/654423051 shutdown_connections 2023-11-24T06:09:30.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.997+0000 7faff6bf0700 1 -- 172.21.15.91:0/654423051 wait complete. 2023-11-24T06:09:30.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.998+0000 7faff6bf0700 1 Processor -- start 2023-11-24T06:09:30.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.998+0000 7faff6bf0700 1 -- start start 2023-11-24T06:09:30.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.999+0000 7faff6bf0700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7faff006afd0 0x7faff0145590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.999+0000 7faff6bf0700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7faff006b3b0 0x7faff01430e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.999+0000 7faff6bf0700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff0072350 0x7faff0143620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.999+0000 7faff6bf0700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff01ea180 con 0x7faff006afd0 2023-11-24T06:09:30.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.999+0000 7faff6bf0700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff01e9570 con 0x7faff0072350 2023-11-24T06:09:30.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.999+0000 7faff6bf0700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff01e9850 con 0x7faff006b3b0 2023-11-24T06:09:30.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.999+0000 7faff498c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7faff006afd0 0x7faff0145590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.008 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.999+0000 7faff498c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7faff006afd0 0x7faff0145590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60480/0 (socket says 172.21.15.91:60480) 2023-11-24T06:09:30.009 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:29.999+0000 7faff498c700 1 -- 172.21.15.91:0/2224569067 learned_addr learned my addr 172.21.15.91:0/2224569067 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:30.009 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.000+0000 7faff518d700 1 --2- 172.21.15.91:0/2224569067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff0072350 0x7faff0143620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.009 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.000+0000 7fafeffff700 1 --2- 172.21.15.91:0/2224569067 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7faff006b3b0 0x7faff01430e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.009 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.000+0000 7faff498c700 1 -- 172.21.15.91:0/2224569067 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7faff006b3b0 msgr2=0x7faff01430e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.009 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.000+0000 7faff498c700 1 --2- 172.21.15.91:0/2224569067 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7faff006b3b0 0x7faff01430e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.009 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.000+0000 7faff498c700 1 -- 172.21.15.91:0/2224569067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff0072350 msgr2=0x7faff0143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.009 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.000+0000 7faff498c700 1 --2- 172.21.15.91:0/2224569067 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7faff0072350 0x7faff0143620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.009 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.000+0000 7faff498c700 1 -- 172.21.15.91:0/2224569067 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe40094a0 con 0x7faff006afd0 2023-11-24T06:09:30.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.000+0000 7faff498c700 1 --2- 172.21.15.91:0/2224569067 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7faff006afd0 0x7faff0145590 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fafe4009fd0 tx=0x7fafe4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:30.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.000+0000 7fafed7fa700 1 -- 172.21.15.91:0/2224569067 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe40036a0 con 0x7faff006afd0 2023-11-24T06:09:30.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.000+0000 7fafed7fa700 1 -- 172.21.15.91:0/2224569067 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe401f070 con 0x7faff006afd0 2023-11-24T06:09:30.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.000+0000 7faff6bf0700 1 -- 172.21.15.91:0/2224569067 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff01e9b30 con 0x7faff006afd0 2023-11-24T06:09:30.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.000+0000 7fafed7fa700 1 -- 172.21.15.91:0/2224569067 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe4003a60 con 0x7faff006afd0 2023-11-24T06:09:30.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.000+0000 7faff6bf0700 1 -- 172.21.15.91:0/2224569067 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff0145ad0 con 0x7faff006afd0 2023-11-24T06:09:30.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.001+0000 7faff6bf0700 1 -- 172.21.15.91:0/2224569067 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faff006b810 con 0x7faff006afd0 2023-11-24T06:09:30.010 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.002+0000 7fafed7fa700 1 -- 172.21.15.91:0/2224569067 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fafe4003bc0 con 0x7faff006afd0 2023-11-24T06:09:30.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.002+0000 7fafed7fa700 1 --2- 172.21.15.91:0/2224569067 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fafd406af90 0x7fafd406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.002+0000 7fafed7fa700 1 -- 172.21.15.91:0/2224569067 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fafe4021070 con 0x7faff006afd0 2023-11-24T06:09:30.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.002+0000 7fafeffff700 1 --2- 172.21.15.91:0/2224569067 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fafd406af90 0x7fafd406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.002+0000 7fafed7fa700 1 --2- 172.21.15.91:0/2224569067 >> [v2:172.21.15.91:6800/1071490516,v1:172.21.15.91:6801/1071490516] conn(0x7fafd4070df0 0x7fafd4073210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.002+0000 7fafed7fa700 1 -- 172.21.15.91:0/2224569067 --> [v2:172.21.15.91:6800/1071490516,v1:172.21.15.91:6801/1071490516] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafd40738e0 con 0x7fafd4070df0 2023-11-24T06:09:30.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.002+0000 7fafed7fa700 1 -- 172.21.15.91:0/2224569067 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafe408f2c0 con 0x7faff006afd0 2023-11-24T06:09:30.011 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.002+0000 7faff518d700 1 --2- 172.21.15.91:0/2224569067 >> [v2:172.21.15.91:6800/1071490516,v1:172.21.15.91:6801/1071490516] conn(0x7fafd4070df0 0x7fafd4073210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.012 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.002+0000 7fafeffff700 1 --2- 172.21.15.91:0/2224569067 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fafd406af90 0x7fafd406d450 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7faff015e750 tx=0x7fafdc0092c0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:30.012 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.003+0000 7faff518d700 1 --2- 172.21.15.91:0/2224569067 >> [v2:172.21.15.91:6800/1071490516,v1:172.21.15.91:6801/1071490516] conn(0x7fafd4070df0 0x7fafd4073210 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:30.012 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.003+0000 7fafed7fa700 1 -- 172.21.15.91:0/2224569067 <== osd.3 v2:172.21.15.91:6800/1071490516 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fafd40738e0 con 0x7fafd4070df0 2023-11-24T06:09:30.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.018+0000 7faff6bf0700 1 -- 172.21.15.91:0/2224569067 --> [v2:172.21.15.91:6800/1071490516,v1:172.21.15.91:6801/1071490516] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7faff015e9b0 con 0x7fafd4070df0 2023-11-24T06:09:30.022 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.018+0000 7fafed7fa700 1 -- 172.21.15.91:0/2224569067 <== osd.3 v2:172.21.15.91:6800/1071490516 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fafe8001f20 con 0x7fafd4070df0 2023-11-24T06:09:30.022 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2023-11-24T06:09:30.023 INFO:teuthology.orchestra.run.smithi091.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T06:09:30.023 INFO:teuthology.orchestra.run.smithi091.stdout:} 2023-11-24T06:09:30.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.018+0000 7faff6bf0700 1 -- 172.21.15.91:0/2224569067 >> [v2:172.21.15.91:6800/1071490516,v1:172.21.15.91:6801/1071490516] conn(0x7fafd4070df0 msgr2=0x7fafd4073210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.018+0000 7faff6bf0700 1 --2- 172.21.15.91:0/2224569067 >> [v2:172.21.15.91:6800/1071490516,v1:172.21.15.91:6801/1071490516] conn(0x7fafd4070df0 0x7fafd4073210 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.018+0000 7faff6bf0700 1 -- 172.21.15.91:0/2224569067 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fafd406af90 msgr2=0x7fafd406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.018+0000 7faff6bf0700 1 --2- 172.21.15.91:0/2224569067 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fafd406af90 0x7fafd406d450 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7faff015e750 tx=0x7fafdc0092c0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.024 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.018+0000 7faff6bf0700 1 -- 172.21.15.91:0/2224569067 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7faff006afd0 msgr2=0x7faff0145590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.018+0000 7faff6bf0700 1 --2- 172.21.15.91:0/2224569067 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7faff006afd0 0x7faff0145590 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fafe4009fd0 tx=0x7fafe4004750 comp rx=0 tx=0).stop 2023-11-24T06:09:30.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.018+0000 7faff518d700 1 -- 172.21.15.91:0/2224569067 reap_dead start 2023-11-24T06:09:30.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.018+0000 7faff6bf0700 1 -- 172.21.15.91:0/2224569067 shutdown_connections 2023-11-24T06:09:30.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.018+0000 7faff6bf0700 1 -- 172.21.15.91:0/2224569067 >> 172.21.15.91:0/2224569067 conn(0x7faff00b2760 msgr2=0x7faff006e880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:30.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.018+0000 7faff6bf0700 1 -- 172.21.15.91:0/2224569067 shutdown_connections 2023-11-24T06:09:30.025 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.018+0000 7faff6bf0700 1 -- 172.21.15.91:0/2224569067 wait complete. 2023-11-24T06:09:30.034 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-11-24T06:09:30.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.151+0000 7f6ac0f8a700 1 Processor -- start 2023-11-24T06:09:30.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.152+0000 7f6ac0f8a700 1 -- start start 2023-11-24T06:09:30.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.152+0000 7f6ac0f8a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6abc15a570 0x7f6abc0ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6ac0f8a700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f6abc0b0490 0x7f6abc0ae900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.158 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6ac0f8a700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc0ae470 con 0x7f6abc06bae0 2023-11-24T06:09:30.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6ac0f8a700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6abc0ad860 con 0x7f6abc0b0490 2023-11-24T06:09:30.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6ac0f8a700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc0adb40 con 0x7f6abc15a570 2023-11-24T06:09:30.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6ab9d9b700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f6abc06bae0 0x7f6abc0afd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55424/0 (socket says 172.21.15.91:55424) 2023-11-24T06:09:30.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6ab9d9b700 1 -- 172.21.15.91:0/1895188023 learned_addr learned my addr 172.21.15.91:0/1895188023 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:30.159 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6abad9d700 1 --2- 172.21.15.91:0/1895188023 >> v2:172.21.15.94:3300/0 conn(0x7f6abc0b0490 0x7f6abc0ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6aba59c700 1 --2- 172.21.15.91:0/1895188023 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6abc15a570 0x7f6abc0ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6ab959a700 1 -- 172.21.15.91:0/1895188023 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789376556 0 0) 0x7f6abc0ae470 con 0x7f6abc06bae0 2023-11-24T06:09:30.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6ab959a700 1 -- 172.21.15.91:0/1895188023 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aa40034b0 con 0x7f6abc06bae0 2023-11-24T06:09:30.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6ab959a700 1 -- 172.21.15.91:0/1895188023 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 204464477 0 0) 0x7f6aa40034b0 con 0x7f6abc06bae0 2023-11-24T06:09:30.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6abad9d700 1 -- 172.21.15.91:0/1895188023 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6abc15a570 msgr2=0x7f6abc0ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:30.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6abad9d700 1 --2- 172.21.15.91:0/1895188023 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6abc15a570 0x7f6abc0ad1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6abad9d700 1 -- 172.21.15.91:0/1895188023 >> v1:172.21.15.91:6789/0 conn(0x7f6abc06bae0 legacy=0x7f6abc0afd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6abad9d700 1 -- 172.21.15.91:0/1895188023 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc0ade20 con 0x7f6abc0b0490 2023-11-24T06:09:30.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6aba59c700 1 --2- 172.21.15.91:0/1895188023 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6abc15a570 0x7f6abc0ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:09:30.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6abad9d700 1 --2- 172.21.15.91:0/1895188023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6abc0b0490 0x7f6abc0ae900 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009d90 tx=0x7f6ab400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dffb74deeb815bb3 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:30.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.153+0000 7f6ab959a700 1 -- 172.21.15.91:0/1895188023 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab400e070 con 0x7f6abc0b0490 2023-11-24T06:09:30.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.154+0000 7f6ab959a700 1 -- 172.21.15.91:0/1895188023 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab4003a20 con 0x7f6abc0b0490 2023-11-24T06:09:30.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.154+0000 7f6ab959a700 1 -- 172.21.15.91:0/1895188023 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac001670 con 0x7f6abc0b0490 2023-11-24T06:09:30.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.154+0000 7f6ac0f8a700 1 -- 172.21.15.91:0/1895188023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6abc0b0490 msgr2=0x7f6abc0ae900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.154+0000 7f6ac0f8a700 1 --2- 172.21.15.91:0/1895188023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6abc0b0490 0x7f6abc0ae900 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009d90 tx=0x7f6ab400b330 comp rx=0 tx=0).stop 2023-11-24T06:09:30.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.154+0000 7f6ac0f8a700 1 -- 172.21.15.91:0/1895188023 shutdown_connections 2023-11-24T06:09:30.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.154+0000 7f6ac0f8a700 1 --2- 172.21.15.91:0/1895188023 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6abc15a570 0x7f6abc0ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.154+0000 7f6ac0f8a700 1 --2- 172.21.15.91:0/1895188023 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6abc0b0490 0x7f6abc0ae900 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.154+0000 7f6ac0f8a700 1 -- 172.21.15.91:0/1895188023 >> 172.21.15.91:0/1895188023 conn(0x7f6abc0717f0 msgr2=0x7f6abc0b5070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:30.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.154+0000 7f6ac0f8a700 1 -- 172.21.15.91:0/1895188023 shutdown_connections 2023-11-24T06:09:30.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.154+0000 7f6ac0f8a700 1 -- 172.21.15.91:0/1895188023 wait complete. 2023-11-24T06:09:30.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.155+0000 7f6ac0f8a700 1 Processor -- start 2023-11-24T06:09:30.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.156+0000 7f6ac0f8a700 1 -- start start 2023-11-24T06:09:30.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.156+0000 7f6ac0f8a700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6abc06bae0 0x7f6abc14ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.156+0000 7f6ac0f8a700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6abc0b0490 0x7f6abc1e4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.156+0000 7f6ac0f8a700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6abc15a570 0x7f6abc154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.156+0000 7f6ac0f8a700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc15b610 con 0x7f6abc06bae0 2023-11-24T06:09:30.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.156+0000 7f6ac0f8a700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc15b8f0 con 0x7f6abc15a570 2023-11-24T06:09:30.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.156+0000 7f6ac0f8a700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc15bbd0 con 0x7f6abc0b0490 2023-11-24T06:09:30.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.156+0000 7f6aba59c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6abc06bae0 0x7f6abc14ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.156+0000 7f6aba59c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6abc06bae0 0x7f6abc14ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60496/0 (socket says 172.21.15.91:60496) 2023-11-24T06:09:30.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.156+0000 7f6aba59c700 1 -- 172.21.15.91:0/3065370770 learned_addr learned my addr 172.21.15.91:0/3065370770 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:30.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.157+0000 7f6aba59c700 1 -- 172.21.15.91:0/3065370770 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6abc0b0490 msgr2=0x7f6abc1e4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.157+0000 7f6aba59c700 1 --2- 172.21.15.91:0/3065370770 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6abc0b0490 0x7f6abc1e4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.157+0000 7f6aba59c700 1 -- 172.21.15.91:0/3065370770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6abc15a570 msgr2=0x7f6abc154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.157+0000 7f6aba59c700 1 --2- 172.21.15.91:0/3065370770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6abc15a570 0x7f6abc154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.157+0000 7f6aba59c700 1 -- 172.21.15.91:0/3065370770 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab000a550 con 0x7f6abc06bae0 2023-11-24T06:09:30.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.157+0000 7f6aba59c700 1 --2- 172.21.15.91:0/3065370770 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6abc06bae0 0x7f6abc14ef70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000f960 tx=0x7f6ab000fd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:30.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.157+0000 7f6aaaffd700 1 -- 172.21.15.91:0/3065370770 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab000a840 con 0x7f6abc06bae0 2023-11-24T06:09:30.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.157+0000 7f6aaaffd700 1 -- 172.21.15.91:0/3065370770 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab000db60 con 0x7f6abc06bae0 2023-11-24T06:09:30.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.157+0000 7f6ac0f8a700 1 -- 172.21.15.91:0/3065370770 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab4009a70 con 0x7f6abc06bae0 2023-11-24T06:09:30.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.157+0000 7f6aaaffd700 1 -- 172.21.15.91:0/3065370770 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0004030 con 0x7f6abc06bae0 2023-11-24T06:09:30.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.157+0000 7f6ac0f8a700 1 -- 172.21.15.91:0/3065370770 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6abc15c190 con 0x7f6abc06bae0 2023-11-24T06:09:30.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.158+0000 7f6ac0f8a700 1 -- 172.21.15.91:0/3065370770 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6abc15c350 con 0x7f6abc06bae0 2023-11-24T06:09:30.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.158+0000 7f6aaaffd700 1 -- 172.21.15.91:0/3065370770 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f6ab0004190 con 0x7f6abc06bae0 2023-11-24T06:09:30.169 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.159+0000 7f6aaaffd700 1 --2- 172.21.15.91:0/3065370770 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6a9c06af60 0x7f6a9c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.169 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.159+0000 7f6ab9d9b700 1 --2- 172.21.15.91:0/3065370770 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6a9c06af60 0x7f6a9c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.169 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.159+0000 7f6aaaffd700 1 -- 172.21.15.91:0/3065370770 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6ab0089510 con 0x7f6abc06bae0 2023-11-24T06:09:30.169 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.159+0000 7f6ab9d9b700 1 --2- 172.21.15.91:0/3065370770 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6a9c06af60 0x7f6a9c06d420 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6aac001eb0 tx=0x7f6aac001a80 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:30.169 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.163+0000 7f6aaaffd700 1 -- 172.21.15.91:0/3065370770 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ab00547d0 con 0x7f6abc06bae0 2023-11-24T06:09:30.356 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.352+0000 7f6ac0f8a700 1 -- 172.21.15.91:0/3065370770 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f6abc15a950 con 0x7f6a9c06af60 2023-11-24T06:09:30.356 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.352+0000 7f6aaaffd700 1 -- 172.21.15.91:0/3065370770 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6aac0070d0 con 0x7f6a9c06af60 2023-11-24T06:09:30.358 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.354+0000 7f6ac0f8a700 1 -- 172.21.15.91:0/3065370770 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6a9c06af60 msgr2=0x7f6a9c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.358 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.354+0000 7f6ac0f8a700 1 --2- 172.21.15.91:0/3065370770 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6a9c06af60 0x7f6a9c06d420 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6aac001eb0 tx=0x7f6aac001a80 comp rx=0 tx=0).stop 2023-11-24T06:09:30.358 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.354+0000 7f6ac0f8a700 1 -- 172.21.15.91:0/3065370770 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6abc06bae0 msgr2=0x7f6abc14ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.354+0000 7f6ac0f8a700 1 --2- 172.21.15.91:0/3065370770 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6abc06bae0 0x7f6abc14ef70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000f960 tx=0x7f6ab000fd20 comp rx=0 tx=0).stop 2023-11-24T06:09:30.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.354+0000 7f6ac0f8a700 1 -- 172.21.15.91:0/3065370770 shutdown_connections 2023-11-24T06:09:30.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.354+0000 7f6ac0f8a700 1 --2- 172.21.15.91:0/3065370770 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f6a9c06af60 0x7f6a9c06d420 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.354+0000 7f6ac0f8a700 1 --2- 172.21.15.91:0/3065370770 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f6abc15a570 0x7f6abc154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.359 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.354+0000 7f6ac0f8a700 1 --2- 172.21.15.91:0/3065370770 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f6abc06bae0 0x7f6abc14ef70 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.360 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.354+0000 7f6ac0f8a700 1 --2- 172.21.15.91:0/3065370770 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f6abc0b0490 0x7f6abc1e4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.360 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.354+0000 7f6ac0f8a700 1 -- 172.21.15.91:0/3065370770 >> 172.21.15.91:0/3065370770 conn(0x7f6abc0717f0 msgr2=0x7f6abc06d780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:30.360 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.354+0000 7f6ac0f8a700 1 -- 172.21.15.91:0/3065370770 shutdown_connections 2023-11-24T06:09:30.360 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.354+0000 7f6ac0f8a700 1 -- 172.21.15.91:0/3065370770 wait complete. 2023-11-24T06:09:30.361 INFO:teuthology.orchestra.run.smithi091.stderr:instructed osd(s) 3 to deep-scrub 2023-11-24T06:09:30.370 INFO:tasks.ceph:Scrubbing osd.4 2023-11-24T06:09:30.370 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-11-24T06:09:30.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.492+0000 7f1a4a484700 1 Processor -- start 2023-11-24T06:09:30.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.492+0000 7f1a4a484700 1 -- start start 2023-11-24T06:09:30.505 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.492+0000 7f1a4a484700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a4406afd0 0x7f1a440b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.505 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.493+0000 7f1a4a484700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f1a4406b3b0 0x7f1a4415e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.506 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.493+0000 7f1a4a484700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a44071ec0 con 0x7f1a44072350 2023-11-24T06:09:30.506 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.493+0000 7f1a4a484700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a44071b90 con 0x7f1a4406b3b0 2023-11-24T06:09:30.506 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.493+0000 7f1a4a484700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4415ed30 con 0x7f1a4406afd0 2023-11-24T06:09:30.506 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.493+0000 7f1a437fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f1a4406b3b0 0x7f1a4415e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.493+0000 7f1a48a21700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f1a44072350 0x7f1a4406dbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55440/0 (socket says 172.21.15.91:55440) 2023-11-24T06:09:30.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.493+0000 7f1a48a21700 1 -- 172.21.15.91:0/3061999652 learned_addr learned my addr 172.21.15.91:0/3061999652 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:30.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.493+0000 7f1a437fe700 1 -- 172.21.15.91:0/3061999652 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a4406afd0 msgr2=0x7f1a440b59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:30.507 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.493+0000 7f1a437fe700 1 --2- 172.21.15.91:0/3061999652 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a4406afd0 0x7f1a440b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.493+0000 7f1a437fe700 1 -- 172.21.15.91:0/3061999652 >> v1:172.21.15.91:6789/0 conn(0x7f1a44072350 legacy=0x7f1a4406dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.493+0000 7f1a437fe700 1 -- 172.21.15.91:0/3061999652 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4415f010 con 0x7f1a4406b3b0 2023-11-24T06:09:30.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.493+0000 7f1a42ffd700 1 -- 172.21.15.91:0/3061999652 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808655641 0 0) 0x7f1a44071ec0 con 0x7f1a44072350 2023-11-24T06:09:30.508 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.493+0000 7f1a437fe700 1 --2- 172.21.15.91:0/3061999652 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a4406b3b0 0x7f1a4415e470 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1a38009840 tx=0x7f1a38009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2309c38621a390 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:30.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.493+0000 7f1a42ffd700 1 -- 172.21.15.91:0/3061999652 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a38011830 con 0x7f1a4406b3b0 2023-11-24T06:09:30.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.493+0000 7f1a42ffd700 1 -- 172.21.15.91:0/3061999652 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a38011990 con 0x7f1a4406b3b0 2023-11-24T06:09:30.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.493+0000 7f1a42ffd700 1 -- 172.21.15.91:0/3061999652 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a38011d20 con 0x7f1a4406b3b0 2023-11-24T06:09:30.509 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.494+0000 7f1a4a484700 1 -- 172.21.15.91:0/3061999652 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a4406b3b0 msgr2=0x7f1a4415e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.494+0000 7f1a4a484700 1 --2- 172.21.15.91:0/3061999652 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a4406b3b0 0x7f1a4415e470 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1a38009840 tx=0x7f1a38009b50 comp rx=0 tx=0).stop 2023-11-24T06:09:30.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.494+0000 7f1a4a484700 1 -- 172.21.15.91:0/3061999652 shutdown_connections 2023-11-24T06:09:30.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.494+0000 7f1a4a484700 1 --2- 172.21.15.91:0/3061999652 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a4406afd0 0x7f1a440b59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.494+0000 7f1a4a484700 1 --2- 172.21.15.91:0/3061999652 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a4406b3b0 0x7f1a4415e470 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.510 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.494+0000 7f1a4a484700 1 -- 172.21.15.91:0/3061999652 >> 172.21.15.91:0/3061999652 conn(0x7f1a440b2760 msgr2=0x7f1a440b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:30.511 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.494+0000 7f1a4a484700 1 -- 172.21.15.91:0/3061999652 shutdown_connections 2023-11-24T06:09:30.511 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.494+0000 7f1a4a484700 1 -- 172.21.15.91:0/3061999652 wait complete. 2023-11-24T06:09:30.511 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.495+0000 7f1a4a484700 1 Processor -- start 2023-11-24T06:09:30.511 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.495+0000 7f1a4a484700 1 -- start start 2023-11-24T06:09:30.512 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a4a484700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a4406afd0 0x7f1a4413de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.512 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a4a484700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a4406b3b0 0x7f1a440ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.512 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a4a484700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a44072350 0x7f1a441609c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.512 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a4a484700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4413f2c0 con 0x7f1a4406b3b0 2023-11-24T06:09:30.513 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a4a484700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a440af530 con 0x7f1a4406afd0 2023-11-24T06:09:30.513 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a4a484700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a440af780 con 0x7f1a44072350 2023-11-24T06:09:30.513 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a437fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a4406b3b0 0x7f1a440ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.513 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a437fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a4406b3b0 0x7f1a440ae270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60508/0 (socket says 172.21.15.91:60508) 2023-11-24T06:09:30.513 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a437fe700 1 -- 172.21.15.91:0/2885624569 learned_addr learned my addr 172.21.15.91:0/2885624569 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:30.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a43fff700 1 --2- 172.21.15.91:0/2885624569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a4406afd0 0x7f1a4413de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a48a21700 1 --2- 172.21.15.91:0/2885624569 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a44072350 0x7f1a441609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a437fe700 1 -- 172.21.15.91:0/2885624569 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a44072350 msgr2=0x7f1a441609c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.514 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a437fe700 1 --2- 172.21.15.91:0/2885624569 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a44072350 0x7f1a441609c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a437fe700 1 -- 172.21.15.91:0/2885624569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a4406afd0 msgr2=0x7f1a4413de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a437fe700 1 --2- 172.21.15.91:0/2885624569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a4406afd0 0x7f1a4413de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a437fe700 1 -- 172.21.15.91:0/2885624569 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a380094a0 con 0x7f1a4406b3b0 2023-11-24T06:09:30.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a48a21700 1 --2- 172.21.15.91:0/2885624569 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1a44072350 0x7f1a441609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:30.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a43fff700 1 --2- 172.21.15.91:0/2885624569 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1a4406afd0 0x7f1a4413de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:30.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.496+0000 7f1a437fe700 1 --2- 172.21.15.91:0/2885624569 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a4406b3b0 0x7f1a440ae270 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1a380090b0 tx=0x7f1a38004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:30.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.497+0000 7f1a40ff9700 1 -- 172.21.15.91:0/2885624569 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a380045f0 con 0x7f1a4406b3b0 2023-11-24T06:09:30.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.497+0000 7f1a40ff9700 1 -- 172.21.15.91:0/2885624569 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3801f070 con 0x7f1a4406b3b0 2023-11-24T06:09:30.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.497+0000 7f1a4a484700 1 -- 172.21.15.91:0/2885624569 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a440afa60 con 0x7f1a4406b3b0 2023-11-24T06:09:30.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.497+0000 7f1a40ff9700 1 -- 172.21.15.91:0/2885624569 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a38003b00 con 0x7f1a4406b3b0 2023-11-24T06:09:30.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.497+0000 7f1a4a484700 1 -- 172.21.15.91:0/2885624569 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a4413f7b0 con 0x7f1a4406b3b0 2023-11-24T06:09:30.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.497+0000 7f1a4a484700 1 -- 172.21.15.91:0/2885624569 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1a4415e9b0 con 0x7f1a4406b3b0 2023-11-24T06:09:30.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.498+0000 7f1a40ff9700 1 -- 172.21.15.91:0/2885624569 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f1a38003c60 con 0x7f1a4406b3b0 2023-11-24T06:09:30.517 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.498+0000 7f1a40ff9700 1 --2- 172.21.15.91:0/2885624569 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1a3006af60 0x7f1a3006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.498+0000 7f1a40ff9700 1 -- 172.21.15.91:0/2885624569 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1a38021070 con 0x7f1a4406b3b0 2023-11-24T06:09:30.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.498+0000 7f1a43fff700 1 --2- 172.21.15.91:0/2885624569 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1a3006af60 0x7f1a3006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.498+0000 7f1a40ff9700 1 --2- 172.21.15.91:0/2885624569 >> [v2:172.21.15.94:6824/3888015416,v1:172.21.15.94:6825/3888015416] conn(0x7f1a30070dc0 0x7f1a300731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.498+0000 7f1a40ff9700 1 -- 172.21.15.91:0/2885624569 --> [v2:172.21.15.94:6824/3888015416,v1:172.21.15.94:6825/3888015416] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a300738b0 con 0x7f1a30070dc0 2023-11-24T06:09:30.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.498+0000 7f1a40ff9700 1 -- 172.21.15.91:0/2885624569 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a3808d270 con 0x7f1a4406b3b0 2023-11-24T06:09:30.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.499+0000 7f1a48a21700 1 --2- 172.21.15.91:0/2885624569 >> [v2:172.21.15.94:6824/3888015416,v1:172.21.15.94:6825/3888015416] conn(0x7f1a30070dc0 0x7f1a300731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.499+0000 7f1a43fff700 1 --2- 172.21.15.91:0/2885624569 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1a3006af60 0x7f1a3006d420 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1a34007900 tx=0x7f1a34008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:30.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.499+0000 7f1a48a21700 1 --2- 172.21.15.91:0/2885624569 >> [v2:172.21.15.94:6824/3888015416,v1:172.21.15.94:6825/3888015416] conn(0x7f1a30070dc0 0x7f1a300731e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:30.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.500+0000 7f1a40ff9700 1 -- 172.21.15.91:0/2885624569 <== osd.4 v2:172.21.15.94:6824/3888015416 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1a300738b0 con 0x7f1a30070dc0 2023-11-24T06:09:30.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.515+0000 7f1a4a484700 1 -- 172.21.15.91:0/2885624569 --> [v2:172.21.15.94:6824/3888015416,v1:172.21.15.94:6825/3888015416] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1a4406b830 con 0x7f1a30070dc0 2023-11-24T06:09:30.520 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.515+0000 7f1a40ff9700 1 -- 172.21.15.91:0/2885624569 <== osd.4 v2:172.21.15.94:6824/3888015416 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f1a3c0070e0 con 0x7f1a30070dc0 2023-11-24T06:09:30.520 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2023-11-24T06:09:30.521 INFO:teuthology.orchestra.run.smithi091.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T06:09:30.521 INFO:teuthology.orchestra.run.smithi091.stdout:} 2023-11-24T06:09:30.522 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.515+0000 7f1a4a484700 1 -- 172.21.15.91:0/2885624569 >> [v2:172.21.15.94:6824/3888015416,v1:172.21.15.94:6825/3888015416] conn(0x7f1a30070dc0 msgr2=0x7f1a300731e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.522 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.515+0000 7f1a4a484700 1 --2- 172.21.15.91:0/2885624569 >> [v2:172.21.15.94:6824/3888015416,v1:172.21.15.94:6825/3888015416] conn(0x7f1a30070dc0 0x7f1a300731e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.522 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.515+0000 7f1a4a484700 1 -- 172.21.15.91:0/2885624569 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1a3006af60 msgr2=0x7f1a3006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.523 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.515+0000 7f1a4a484700 1 --2- 172.21.15.91:0/2885624569 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1a3006af60 0x7f1a3006d420 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1a34007900 tx=0x7f1a34008040 comp rx=0 tx=0).stop 2023-11-24T06:09:30.523 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.515+0000 7f1a4a484700 1 -- 172.21.15.91:0/2885624569 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a4406b3b0 msgr2=0x7f1a440ae270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.523 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.515+0000 7f1a4a484700 1 --2- 172.21.15.91:0/2885624569 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1a4406b3b0 0x7f1a440ae270 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1a380090b0 tx=0x7f1a38004750 comp rx=0 tx=0).stop 2023-11-24T06:09:30.523 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.516+0000 7f1a48a21700 1 -- 172.21.15.91:0/2885624569 reap_dead start 2023-11-24T06:09:30.523 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.516+0000 7f1a4a484700 1 -- 172.21.15.91:0/2885624569 shutdown_connections 2023-11-24T06:09:30.524 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.516+0000 7f1a4a484700 1 -- 172.21.15.91:0/2885624569 >> 172.21.15.91:0/2885624569 conn(0x7f1a440b2760 msgr2=0x7f1a44144a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:30.524 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.516+0000 7f1a4a484700 1 -- 172.21.15.91:0/2885624569 shutdown_connections 2023-11-24T06:09:30.524 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.516+0000 7f1a4a484700 1 -- 172.21.15.91:0/2885624569 wait complete. 2023-11-24T06:09:30.531 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-11-24T06:09:30.652 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.648+0000 7f8e2e8bf700 1 Processor -- start 2023-11-24T06:09:30.652 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.648+0000 7f8e2e8bf700 1 -- start start 2023-11-24T06:09:30.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.649+0000 7f8e2e8bf700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f8e280b4150 0x7f8e2806de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.649+0000 7f8e2e8bf700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e28071f50 con 0x7f8e280b5ab0 2023-11-24T06:09:30.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.649+0000 7f8e2e8bf700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e28072230 con 0x7f8e2806b620 2023-11-24T06:09:30.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.649+0000 7f8e2e8bf700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e28072510 con 0x7f8e280b4150 2023-11-24T06:09:30.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.649+0000 7f8e277fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f8e280b4150 0x7f8e2806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.649+0000 7f8e2ce5c700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f8e280b5ab0 0x7f8e2806e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55456/0 (socket says 172.21.15.91:55456) 2023-11-24T06:09:30.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.649+0000 7f8e2ce5c700 1 -- 172.21.15.91:0/1530705346 learned_addr learned my addr 172.21.15.91:0/1530705346 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:30.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.649+0000 7f8e277fe700 1 -- 172.21.15.91:0/1530705346 >> v1:172.21.15.94:6789/0 conn(0x7f8e2806b620 legacy=0x7f8e280b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.650+0000 7f8e277fe700 1 -- 172.21.15.91:0/1530705346 >> v1:172.21.15.91:6789/0 conn(0x7f8e280b5ab0 legacy=0x7f8e2806e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.649+0000 7f8e26ffd700 1 -- 172.21.15.91:0/1530705346 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485858428 0 0) 0x7f8e28071f50 con 0x7f8e280b5ab0 2023-11-24T06:09:30.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.650+0000 7f8e277fe700 1 -- 172.21.15.91:0/1530705346 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e280727f0 con 0x7f8e280b4150 2023-11-24T06:09:30.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.650+0000 7f8e277fe700 1 --2- 172.21.15.91:0/1530705346 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e280b4150 0x7f8e2806de50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009840 tx=0x7f8e18009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=725e94bd7575b4d6 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:30.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.650+0000 7f8e26ffd700 1 -- 172.21.15.91:0/1530705346 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e18011830 con 0x7f8e280b4150 2023-11-24T06:09:30.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.650+0000 7f8e26ffd700 1 -- 172.21.15.91:0/1530705346 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e18011990 con 0x7f8e280b4150 2023-11-24T06:09:30.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.650+0000 7f8e26ffd700 1 -- 172.21.15.91:0/1530705346 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e18011d20 con 0x7f8e280b4150 2023-11-24T06:09:30.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.650+0000 7f8e2e8bf700 1 -- 172.21.15.91:0/1530705346 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e280b4150 msgr2=0x7f8e2806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.657 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.650+0000 7f8e2e8bf700 1 --2- 172.21.15.91:0/1530705346 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e280b4150 0x7f8e2806de50 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009840 tx=0x7f8e18009b50 comp rx=0 tx=0).stop 2023-11-24T06:09:30.657 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.650+0000 7f8e2e8bf700 1 -- 172.21.15.91:0/1530705346 shutdown_connections 2023-11-24T06:09:30.657 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.650+0000 7f8e2e8bf700 1 --2- 172.21.15.91:0/1530705346 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e280b4150 0x7f8e2806de50 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.657 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.650+0000 7f8e2e8bf700 1 -- 172.21.15.91:0/1530705346 >> 172.21.15.91:0/1530705346 conn(0x7f8e280b2740 msgr2=0x7f8e280b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:30.657 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.650+0000 7f8e2e8bf700 1 -- 172.21.15.91:0/1530705346 shutdown_connections 2023-11-24T06:09:30.657 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.650+0000 7f8e2e8bf700 1 -- 172.21.15.91:0/1530705346 wait complete. 2023-11-24T06:09:30.660 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.652+0000 7f8e2e8bf700 1 Processor -- start 2023-11-24T06:09:30.660 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.652+0000 7f8e2e8bf700 1 -- start start 2023-11-24T06:09:30.660 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.652+0000 7f8e2e8bf700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e2806b620 0x7f8e28135b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e2e8bf700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8e280b4150 0x7f8e280cb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e2e8bf700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e280b5ab0 0x7f8e28133010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e2e8bf700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e280c8a10 con 0x7f8e2806b620 2023-11-24T06:09:30.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e2e8bf700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e280c8c60 con 0x7f8e280b5ab0 2023-11-24T06:09:30.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e2e8bf700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e280c8f40 con 0x7f8e280b4150 2023-11-24T06:09:30.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e27fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e2806b620 0x7f8e28135b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e27fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e2806b620 0x7f8e28135b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60524/0 (socket says 172.21.15.91:60524) 2023-11-24T06:09:30.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e27fff700 1 -- 172.21.15.91:0/1275469131 learned_addr learned my addr 172.21.15.91:0/1275469131 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:30.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e2ce5c700 1 --2- 172.21.15.91:0/1275469131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e280b5ab0 0x7f8e28133010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e277fe700 1 --2- 172.21.15.91:0/1275469131 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8e280b4150 0x7f8e280cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e27fff700 1 -- 172.21.15.91:0/1275469131 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8e280b4150 msgr2=0x7f8e280cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e27fff700 1 --2- 172.21.15.91:0/1275469131 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8e280b4150 0x7f8e280cb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e27fff700 1 -- 172.21.15.91:0/1275469131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e280b5ab0 msgr2=0x7f8e28133010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e27fff700 1 --2- 172.21.15.91:0/1275469131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e280b5ab0 0x7f8e28133010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e27fff700 1 -- 172.21.15.91:0/1275469131 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e180094a0 con 0x7f8e2806b620 2023-11-24T06:09:30.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e2ce5c700 1 --2- 172.21.15.91:0/1275469131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e280b5ab0 0x7f8e28133010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:30.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e27fff700 1 --2- 172.21.15.91:0/1275469131 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e2806b620 0x7f8e28135b90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00c4e0 tx=0x7f8e1c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:30.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e24ff9700 1 -- 172.21.15.91:0/1275469131 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c0052d0 con 0x7f8e2806b620 2023-11-24T06:09:30.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e24ff9700 1 -- 172.21.15.91:0/1275469131 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1c005430 con 0x7f8e2806b620 2023-11-24T06:09:30.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e24ff9700 1 -- 172.21.15.91:0/1275469131 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c0038c0 con 0x7f8e2806b620 2023-11-24T06:09:30.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e2e8bf700 1 -- 172.21.15.91:0/1275469131 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e280c9220 con 0x7f8e2806b620 2023-11-24T06:09:30.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.653+0000 7f8e2e8bf700 1 -- 172.21.15.91:0/1275469131 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e28133550 con 0x7f8e2806b620 2023-11-24T06:09:30.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.654+0000 7f8e2e8bf700 1 -- 172.21.15.91:0/1275469131 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e28070130 con 0x7f8e2806b620 2023-11-24T06:09:30.664 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.655+0000 7f8e24ff9700 1 -- 172.21.15.91:0/1275469131 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f8e1c003a20 con 0x7f8e2806b620 2023-11-24T06:09:30.664 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.655+0000 7f8e24ff9700 1 --2- 172.21.15.91:0/1275469131 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8e1406af60 0x7f8e1406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.664 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.655+0000 7f8e24ff9700 1 -- 172.21.15.91:0/1275469131 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8e1c0887e0 con 0x7f8e2806b620 2023-11-24T06:09:30.664 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.655+0000 7f8e277fe700 1 --2- 172.21.15.91:0/1275469131 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8e1406af60 0x7f8e1406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.664 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.655+0000 7f8e277fe700 1 --2- 172.21.15.91:0/1275469131 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8e1406af60 0x7f8e1406d420 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009b90 tx=0x7f8e1801c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:30.665 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.660+0000 7f8e24ff9700 1 -- 172.21.15.91:0/1275469131 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e1c053ba0 con 0x7f8e2806b620 2023-11-24T06:09:30.854 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.850+0000 7f8e2e8bf700 1 -- 172.21.15.91:0/1275469131 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e2806ba00 con 0x7f8e1406af60 2023-11-24T06:09:30.854 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.851+0000 7f8e24ff9700 1 -- 172.21.15.91:0/1275469131 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8e18011b00 con 0x7f8e1406af60 2023-11-24T06:09:30.856 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.852+0000 7f8e2e8bf700 1 -- 172.21.15.91:0/1275469131 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8e1406af60 msgr2=0x7f8e1406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.852+0000 7f8e2e8bf700 1 --2- 172.21.15.91:0/1275469131 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8e1406af60 0x7f8e1406d420 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009b90 tx=0x7f8e1801c040 comp rx=0 tx=0).stop 2023-11-24T06:09:30.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.852+0000 7f8e2e8bf700 1 -- 172.21.15.91:0/1275469131 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e2806b620 msgr2=0x7f8e28135b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.852+0000 7f8e2e8bf700 1 --2- 172.21.15.91:0/1275469131 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e2806b620 0x7f8e28135b90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00c4e0 tx=0x7f8e1c00c8a0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.852+0000 7f8e2e8bf700 1 -- 172.21.15.91:0/1275469131 shutdown_connections 2023-11-24T06:09:30.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.852+0000 7f8e2e8bf700 1 --2- 172.21.15.91:0/1275469131 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f8e1406af60 0x7f8e1406d420 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.852+0000 7f8e2e8bf700 1 --2- 172.21.15.91:0/1275469131 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f8e280b5ab0 0x7f8e28133010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.852+0000 7f8e2e8bf700 1 --2- 172.21.15.91:0/1275469131 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f8e2806b620 0x7f8e28135b90 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.857 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.852+0000 7f8e2e8bf700 1 --2- 172.21.15.91:0/1275469131 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f8e280b4150 0x7f8e280cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.852+0000 7f8e2e8bf700 1 -- 172.21.15.91:0/1275469131 >> 172.21.15.91:0/1275469131 conn(0x7f8e280b2740 msgr2=0x7f8e28146550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:30.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.853+0000 7f8e2e8bf700 1 -- 172.21.15.91:0/1275469131 shutdown_connections 2023-11-24T06:09:30.858 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.853+0000 7f8e2e8bf700 1 -- 172.21.15.91:0/1275469131 wait complete. 2023-11-24T06:09:30.858 INFO:teuthology.orchestra.run.smithi091.stderr:instructed osd(s) 4 to deep-scrub 2023-11-24T06:09:30.868 INFO:tasks.ceph:Scrubbing osd.5 2023-11-24T06:09:30.868 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-11-24T06:09:30.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.988+0000 7eff56320700 1 Processor -- start 2023-11-24T06:09:30.992 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.988+0000 7eff56320700 1 -- start start 2023-11-24T06:09:30.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.989+0000 7eff56320700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7eff5015dd30 0x7eff5015e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.989+0000 7eff56320700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7eff500d2490 0x7eff500d28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:30.995 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.990+0000 7eff56320700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff500d22b0 con 0x7eff500cea30 2023-11-24T06:09:30.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.990+0000 7eff56320700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff500d2e20 con 0x7eff500d2490 2023-11-24T06:09:30.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.990+0000 7eff56320700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff5015e650 con 0x7eff5015dd30 2023-11-24T06:09:30.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.990+0000 7eff548bd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7eff500d2490 0x7eff500d28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.990+0000 7eff548bd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7eff500d2490 0x7eff500d28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60536/0 (socket says 172.21.15.91:60536) 2023-11-24T06:09:30.996 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.990+0000 7eff548bd700 1 -- 172.21.15.91:0/2839426945 learned_addr learned my addr 172.21.15.91:0/2839426945 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:30.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.990+0000 7eff4ffff700 1 --2- 172.21.15.91:0/2839426945 >> v2:172.21.15.94:3300/0 conn(0x7eff5015dd30 0x7eff5015e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:30.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.990+0000 7eff548bd700 1 -- 172.21.15.91:0/2839426945 >> v2:172.21.15.94:3300/0 conn(0x7eff5015dd30 msgr2=0x7eff5015e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:30.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.990+0000 7eff548bd700 1 --2- 172.21.15.91:0/2839426945 >> v2:172.21.15.94:3300/0 conn(0x7eff5015dd30 0x7eff5015e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.990+0000 7eff548bd700 1 -- 172.21.15.91:0/2839426945 >> v1:172.21.15.94:6789/0 conn(0x7eff500cea30 legacy=0x7eff500d1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.990+0000 7eff548bd700 1 -- 172.21.15.91:0/2839426945 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff500cef10 con 0x7eff500d2490 2023-11-24T06:09:30.997 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.990+0000 7eff548bd700 1 --2- 172.21.15.91:0/2839426945 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff500d2490 0x7eff500d28e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7eff5006b170 tx=0x7eff4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4feece8c4ef83cb server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:30.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.990+0000 7eff4effd700 1 -- 172.21.15.91:0/2839426945 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4800e070 con 0x7eff500d2490 2023-11-24T06:09:30.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.990+0000 7eff4effd700 1 -- 172.21.15.91:0/2839426945 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4800b910 con 0x7eff500d2490 2023-11-24T06:09:30.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.990+0000 7eff4effd700 1 -- 172.21.15.91:0/2839426945 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4800bcc0 con 0x7eff500d2490 2023-11-24T06:09:30.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.991+0000 7eff56320700 1 -- 172.21.15.91:0/2839426945 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff500d2490 msgr2=0x7eff500d28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:30.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.991+0000 7eff56320700 1 --2- 172.21.15.91:0/2839426945 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff500d2490 0x7eff500d28e0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7eff5006b170 tx=0x7eff4800b330 comp rx=0 tx=0).stop 2023-11-24T06:09:30.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.991+0000 7eff56320700 1 -- 172.21.15.91:0/2839426945 shutdown_connections 2023-11-24T06:09:30.998 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.991+0000 7eff56320700 1 --2- 172.21.15.91:0/2839426945 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff500d2490 0x7eff500d28e0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.991+0000 7eff56320700 1 --2- 172.21.15.91:0/2839426945 >> v2:172.21.15.94:3300/0 conn(0x7eff5015dd30 0x7eff5015e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:30.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.991+0000 7eff56320700 1 -- 172.21.15.91:0/2839426945 >> 172.21.15.91:0/2839426945 conn(0x7eff5009d110 msgr2=0x7eff500d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:30.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.991+0000 7eff56320700 1 -- 172.21.15.91:0/2839426945 shutdown_connections 2023-11-24T06:09:30.999 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.991+0000 7eff56320700 1 -- 172.21.15.91:0/2839426945 wait complete. 2023-11-24T06:09:31.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.992+0000 7eff56320700 1 Processor -- start 2023-11-24T06:09:31.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.992+0000 7eff56320700 1 -- start start 2023-11-24T06:09:31.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.993+0000 7eff56320700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff500cea30 0x7eff500b5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.002 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.993+0000 7eff56320700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff500d2490 0x7eff500cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.993+0000 7eff56320700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff5015dd30 0x7eff5015bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.993+0000 7eff56320700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff500a0820 con 0x7eff500d2490 2023-11-24T06:09:31.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.993+0000 7eff56320700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff500ccbc0 con 0x7eff500cea30 2023-11-24T06:09:31.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.993+0000 7eff56320700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5009eb30 con 0x7eff5015dd30 2023-11-24T06:09:31.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.993+0000 7eff4f7fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff500d2490 0x7eff500cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.993+0000 7eff4f7fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff500d2490 0x7eff500cc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60540/0 (socket says 172.21.15.91:60540) 2023-11-24T06:09:31.003 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.993+0000 7eff4f7fe700 1 -- 172.21.15.91:0/1271213028 learned_addr learned my addr 172.21.15.91:0/1271213028 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:31.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.993+0000 7eff4ffff700 1 --2- 172.21.15.91:0/1271213028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff500cea30 0x7eff500b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.993+0000 7eff548bd700 1 --2- 172.21.15.91:0/1271213028 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff5015dd30 0x7eff5015bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.993+0000 7eff4f7fe700 1 -- 172.21.15.91:0/1271213028 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff5015dd30 msgr2=0x7eff5015bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.993+0000 7eff4f7fe700 1 --2- 172.21.15.91:0/1271213028 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7eff5015dd30 0x7eff5015bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.993+0000 7eff4f7fe700 1 -- 172.21.15.91:0/1271213028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff500cea30 msgr2=0x7eff500b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.004 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.993+0000 7eff4f7fe700 1 --2- 172.21.15.91:0/1271213028 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7eff500cea30 0x7eff500b5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.005 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.993+0000 7eff4f7fe700 1 -- 172.21.15.91:0/1271213028 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff48009b30 con 0x7eff500d2490 2023-11-24T06:09:31.005 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.994+0000 7eff4f7fe700 1 --2- 172.21.15.91:0/1271213028 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff500d2490 0x7eff500cc680 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7eff4000c420 tx=0x7eff4000c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:31.005 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.994+0000 7eff4cff9700 1 -- 172.21.15.91:0/1271213028 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff400052d0 con 0x7eff500d2490 2023-11-24T06:09:31.005 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.994+0000 7eff4cff9700 1 -- 172.21.15.91:0/1271213028 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff40005430 con 0x7eff500d2490 2023-11-24T06:09:31.005 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.994+0000 7eff56320700 1 -- 172.21.15.91:0/1271213028 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff5009ee10 con 0x7eff500d2490 2023-11-24T06:09:31.005 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.994+0000 7eff4cff9700 1 -- 172.21.15.91:0/1271213028 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff400038c0 con 0x7eff500d2490 2023-11-24T06:09:31.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.994+0000 7eff56320700 1 -- 172.21.15.91:0/1271213028 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff500a0ac0 con 0x7eff500d2490 2023-11-24T06:09:31.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.994+0000 7eff56320700 1 -- 172.21.15.91:0/1271213028 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7eff500d3a10 con 0x7eff500d2490 2023-11-24T06:09:31.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.995+0000 7eff4cff9700 1 -- 172.21.15.91:0/1271213028 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7eff40003a20 con 0x7eff500d2490 2023-11-24T06:09:31.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.995+0000 7eff4cff9700 1 --2- 172.21.15.91:0/1271213028 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7eff3c06af60 0x7eff3c06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.996+0000 7eff4cff9700 1 -- 172.21.15.91:0/1271213028 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7eff40088660 con 0x7eff500d2490 2023-11-24T06:09:31.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.996+0000 7eff4ffff700 1 --2- 172.21.15.91:0/1271213028 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7eff3c06af60 0x7eff3c06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.006 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.996+0000 7eff4cff9700 1 --2- 172.21.15.91:0/1271213028 >> [v2:172.21.15.94:6808/4188708142,v1:172.21.15.94:6809/4188708142] conn(0x7eff3c070dc0 0x7eff3c0731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.996+0000 7eff4cff9700 1 -- 172.21.15.91:0/1271213028 --> [v2:172.21.15.94:6808/4188708142,v1:172.21.15.94:6809/4188708142] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff3c0738b0 con 0x7eff3c070dc0 2023-11-24T06:09:31.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.996+0000 7eff4cff9700 1 -- 172.21.15.91:0/1271213028 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff4008ad10 con 0x7eff500d2490 2023-11-24T06:09:31.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.996+0000 7eff548bd700 1 --2- 172.21.15.91:0/1271213028 >> [v2:172.21.15.94:6808/4188708142,v1:172.21.15.94:6809/4188708142] conn(0x7eff3c070dc0 0x7eff3c0731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.996+0000 7eff4ffff700 1 --2- 172.21.15.91:0/1271213028 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7eff3c06af60 0x7eff3c06d420 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7eff44009be0 tx=0x7eff44009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:31.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.996+0000 7eff548bd700 1 --2- 172.21.15.91:0/1271213028 >> [v2:172.21.15.94:6808/4188708142,v1:172.21.15.94:6809/4188708142] conn(0x7eff3c070dc0 0x7eff3c0731e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:31.007 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:30.997+0000 7eff4cff9700 1 -- 172.21.15.91:0/1271213028 <== osd.5 v2:172.21.15.94:6808/4188708142 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7eff3c0738b0 con 0x7eff3c070dc0 2023-11-24T06:09:31.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.012+0000 7eff56320700 1 -- 172.21.15.91:0/1271213028 --> [v2:172.21.15.94:6808/4188708142,v1:172.21.15.94:6809/4188708142] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7eff500cf7b0 con 0x7eff3c070dc0 2023-11-24T06:09:31.016 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.013+0000 7eff4cff9700 1 -- 172.21.15.91:0/1271213028 <== osd.5 v2:172.21.15.94:6808/4188708142 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7eff480092e0 con 0x7eff3c070dc0 2023-11-24T06:09:31.017 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2023-11-24T06:09:31.017 INFO:teuthology.orchestra.run.smithi091.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T06:09:31.017 INFO:teuthology.orchestra.run.smithi091.stdout:} 2023-11-24T06:09:31.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.013+0000 7eff56320700 1 -- 172.21.15.91:0/1271213028 >> [v2:172.21.15.94:6808/4188708142,v1:172.21.15.94:6809/4188708142] conn(0x7eff3c070dc0 msgr2=0x7eff3c0731e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.013+0000 7eff56320700 1 --2- 172.21.15.91:0/1271213028 >> [v2:172.21.15.94:6808/4188708142,v1:172.21.15.94:6809/4188708142] conn(0x7eff3c070dc0 0x7eff3c0731e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.013+0000 7eff56320700 1 -- 172.21.15.91:0/1271213028 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7eff3c06af60 msgr2=0x7eff3c06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.013+0000 7eff56320700 1 --2- 172.21.15.91:0/1271213028 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7eff3c06af60 0x7eff3c06d420 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7eff44009be0 tx=0x7eff44009210 comp rx=0 tx=0).stop 2023-11-24T06:09:31.018 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.013+0000 7eff56320700 1 -- 172.21.15.91:0/1271213028 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff500d2490 msgr2=0x7eff500cc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.013+0000 7eff56320700 1 --2- 172.21.15.91:0/1271213028 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7eff500d2490 0x7eff500cc680 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7eff4000c420 tx=0x7eff4000c730 comp rx=0 tx=0).stop 2023-11-24T06:09:31.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.013+0000 7eff548bd700 1 -- 172.21.15.91:0/1271213028 reap_dead start 2023-11-24T06:09:31.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.013+0000 7eff56320700 1 -- 172.21.15.91:0/1271213028 shutdown_connections 2023-11-24T06:09:31.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.013+0000 7eff56320700 1 -- 172.21.15.91:0/1271213028 >> 172.21.15.91:0/1271213028 conn(0x7eff5009d110 msgr2=0x7eff500d3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:31.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.013+0000 7eff56320700 1 -- 172.21.15.91:0/1271213028 shutdown_connections 2023-11-24T06:09:31.019 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.013+0000 7eff56320700 1 -- 172.21.15.91:0/1271213028 wait complete. 2023-11-24T06:09:31.028 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-11-24T06:09:31.150 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.146+0000 7fa1f3036700 1 Processor -- start 2023-11-24T06:09:31.151 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.146+0000 7fa1f3036700 1 -- start start 2023-11-24T06:09:31.153 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.147+0000 7fa1f3036700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa1ec06b620 0x7fa1ec0b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.153 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.147+0000 7fa1f3036700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa1ec06ee60 0x7fa1ec0b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.153 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.147+0000 7fa1f3036700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec071f50 con 0x7fa1ec0b5ab0 2023-11-24T06:09:31.153 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.147+0000 7fa1f3036700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ec072230 con 0x7fa1ec06b620 2023-11-24T06:09:31.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.147+0000 7fa1f3036700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec072510 con 0x7fa1ec06ee60 2023-11-24T06:09:31.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.148+0000 7fa1f0dd2700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa1ec06b620 0x7fa1ec0b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.148+0000 7fa1f15d3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa1ec06ee60 0x7fa1ec0b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.148+0000 7fa1f0dd2700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa1ec06b620 0x7fa1ec0b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3300/0 says I am v2:172.21.15.91:44940/0 (socket says 172.21.15.91:44940) 2023-11-24T06:09:31.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.148+0000 7fa1ebfff700 1 --1- >> v1:172.21.15.94:6789/0 conn(0x7fa1ec0b5ab0 0x7fa1ec161ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.94:6789/0 says I am v1:172.21.15.91:57274/0 (socket says 172.21.15.91:57274) 2023-11-24T06:09:31.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.148+0000 7fa1f15d3700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa1ec06ee60 0x7fa1ec0b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:43984/0 (socket says 172.21.15.91:43984) 2023-11-24T06:09:31.154 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.148+0000 7fa1f0dd2700 1 -- 172.21.15.91:0/816787953 learned_addr learned my addr 172.21.15.91:0/816787953 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:31.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.148+0000 7fa1f0dd2700 1 -- 172.21.15.91:0/816787953 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa1ec06ee60 msgr2=0x7fa1ec0b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:31.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.148+0000 7fa1f0dd2700 1 --2- 172.21.15.91:0/816787953 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa1ec06ee60 0x7fa1ec0b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.148+0000 7fa1f0dd2700 1 -- 172.21.15.91:0/816787953 >> v1:172.21.15.94:6789/0 conn(0x7fa1ec0b5ab0 legacy=0x7fa1ec161ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.148+0000 7fa1f0dd2700 1 -- 172.21.15.91:0/816787953 --> v2:172.21.15.94:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec0727f0 con 0x7fa1ec06b620 2023-11-24T06:09:31.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.148+0000 7fa1eb7fe700 1 -- 172.21.15.91:0/816787953 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947060943 0 0) 0x7fa1ec071f50 con 0x7fa1ec0b5ab0 2023-11-24T06:09:31.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.148+0000 7fa1f0dd2700 1 --2- 172.21.15.91:0/816787953 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ec06b620 0x7fa1ec0b3a40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00098f0 tx=0x7fa1e0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf221490d06389df server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:31.155 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.148+0000 7fa1eb7fe700 1 -- 172.21.15.91:0/816787953 <== mon.1 v2:172.21.15.94:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0011830 con 0x7fa1ec06b620 2023-11-24T06:09:31.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.148+0000 7fa1eb7fe700 1 -- 172.21.15.91:0/816787953 <== mon.1 v2:172.21.15.94:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e0011990 con 0x7fa1ec06b620 2023-11-24T06:09:31.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.148+0000 7fa1eb7fe700 1 -- 172.21.15.91:0/816787953 <== mon.1 v2:172.21.15.94:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0011d20 con 0x7fa1ec06b620 2023-11-24T06:09:31.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.149+0000 7fa1f3036700 1 -- 172.21.15.91:0/816787953 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ec06b620 msgr2=0x7fa1ec0b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.149+0000 7fa1f3036700 1 --2- 172.21.15.91:0/816787953 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ec06b620 0x7fa1ec0b3a40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00098f0 tx=0x7fa1e0009c00 comp rx=0 tx=0).stop 2023-11-24T06:09:31.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.149+0000 7fa1f3036700 1 -- 172.21.15.91:0/816787953 shutdown_connections 2023-11-24T06:09:31.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.149+0000 7fa1f3036700 1 --2- 172.21.15.91:0/816787953 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa1ec06ee60 0x7fa1ec0b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.149+0000 7fa1f3036700 1 --2- 172.21.15.91:0/816787953 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ec06b620 0x7fa1ec0b3a40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.156 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.149+0000 7fa1f3036700 1 -- 172.21.15.91:0/816787953 >> 172.21.15.91:0/816787953 conn(0x7fa1ec0b2740 msgr2=0x7fa1ec0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:31.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.149+0000 7fa1f3036700 1 -- 172.21.15.91:0/816787953 shutdown_connections 2023-11-24T06:09:31.157 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.149+0000 7fa1f3036700 1 -- 172.21.15.91:0/816787953 wait complete. 2023-11-24T06:09:31.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.150+0000 7fa1f3036700 1 Processor -- start 2023-11-24T06:09:31.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.150+0000 7fa1f3036700 1 -- start start 2023-11-24T06:09:31.160 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.151+0000 7fa1f3036700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa1ec06ee60 0x7fa1ec0c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.151+0000 7fa1f3036700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa1ec0b5ab0 0x7fa1ec141fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.151+0000 7fa1f3036700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ec0c70c0 0x7fa1ec1e7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.151+0000 7fa1f3036700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0c6020 con 0x7fa1ec06ee60 2023-11-24T06:09:31.161 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.151+0000 7fa1f3036700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0c6300 con 0x7fa1ec0c70c0 2023-11-24T06:09:31.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.151+0000 7fa1f3036700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0c65e0 con 0x7fa1ec0b5ab0 2023-11-24T06:09:31.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.151+0000 7fa1f0dd2700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa1ec06ee60 0x7fa1ec0c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.151+0000 7fa1ebfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa1ec0b5ab0 0x7fa1ec141fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.162 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.151+0000 7fa1ebfff700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa1ec0b5ab0 0x7fa1ec141fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.94:3301/0 says I am v2:172.21.15.91:43988/0 (socket says 172.21.15.91:43988) 2023-11-24T06:09:31.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.151+0000 7fa1ebfff700 1 -- 172.21.15.91:0/1746145039 learned_addr learned my addr 172.21.15.91:0/1746145039 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:31.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.151+0000 7fa1f15d3700 1 --2- 172.21.15.91:0/1746145039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ec0c70c0 0x7fa1ec1e7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.152+0000 7fa1ebfff700 1 -- 172.21.15.91:0/1746145039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ec0c70c0 msgr2=0x7fa1ec1e7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.163 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.152+0000 7fa1ebfff700 1 --2- 172.21.15.91:0/1746145039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ec0c70c0 0x7fa1ec1e7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.152+0000 7fa1ebfff700 1 -- 172.21.15.91:0/1746145039 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa1ec06ee60 msgr2=0x7fa1ec0c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.152+0000 7fa1ebfff700 1 --2- 172.21.15.91:0/1746145039 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa1ec06ee60 0x7fa1ec0c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.152+0000 7fa1ebfff700 1 -- 172.21.15.91:0/1746145039 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e0009550 con 0x7fa1ec0b5ab0 2023-11-24T06:09:31.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.152+0000 7fa1f15d3700 1 --2- 172.21.15.91:0/1746145039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ec0c70c0 0x7fa1ec1e7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-24T06:09:31.164 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.152+0000 7fa1ebfff700 1 --2- 172.21.15.91:0/1746145039 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa1ec0b5ab0 0x7fa1ec141fc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800c520 tx=0x7fa1d800c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:31.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.152+0000 7fa1e97fa700 1 -- 172.21.15.91:0/1746145039 <== mon.2 v2:172.21.15.94:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8004c80 con 0x7fa1ec0b5ab0 2023-11-24T06:09:31.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.152+0000 7fa1e97fa700 1 -- 172.21.15.91:0/1746145039 <== mon.2 v2:172.21.15.94:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d8004de0 con 0x7fa1ec0b5ab0 2023-11-24T06:09:31.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.152+0000 7fa1f3036700 1 -- 172.21.15.91:0/1746145039 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ec0c68c0 con 0x7fa1ec0b5ab0 2023-11-24T06:09:31.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.152+0000 7fa1e97fa700 1 -- 172.21.15.91:0/1746145039 <== mon.2 v2:172.21.15.94:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d80072d0 con 0x7fa1ec0b5ab0 2023-11-24T06:09:31.165 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.152+0000 7fa1f3036700 1 -- 172.21.15.91:0/1746145039 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ec0c7550 con 0x7fa1ec0b5ab0 2023-11-24T06:09:31.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.153+0000 7fa1f3036700 1 -- 172.21.15.91:0/1746145039 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec068180 con 0x7fa1ec0b5ab0 2023-11-24T06:09:31.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.153+0000 7fa1e97fa700 1 -- 172.21.15.91:0/1746145039 <== mon.2 v2:172.21.15.94:3301/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fa1d8007430 con 0x7fa1ec0b5ab0 2023-11-24T06:09:31.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.154+0000 7fa1e97fa700 1 --2- 172.21.15.91:0/1746145039 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa1d006af60 0x7fa1d006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.166 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.154+0000 7fa1e97fa700 1 -- 172.21.15.91:0/1746145039 <== mon.2 v2:172.21.15.94:3301/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa1d8088e60 con 0x7fa1ec0b5ab0 2023-11-24T06:09:31.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.154+0000 7fa1f0dd2700 1 --2- 172.21.15.91:0/1746145039 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa1d006af60 0x7fa1d006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.167 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.154+0000 7fa1f0dd2700 1 --2- 172.21.15.91:0/1746145039 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa1d006af60 0x7fa1d006d420 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec06da40 tx=0x7fa1e0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:31.168 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.159+0000 7fa1e97fa700 1 -- 172.21.15.91:0/1746145039 <== mon.2 v2:172.21.15.94:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa1d8054230 con 0x7fa1ec0b5ab0 2023-11-24T06:09:31.353 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.349+0000 7fa1f3036700 1 -- 172.21.15.91:0/1746145039 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1ec06ba00 con 0x7fa1d006af60 2023-11-24T06:09:31.353 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.349+0000 7fa1e97fa700 1 -- 172.21.15.91:0/1746145039 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa1e0011b00 con 0x7fa1d006af60 2023-11-24T06:09:31.356 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.351+0000 7fa1f3036700 1 -- 172.21.15.91:0/1746145039 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa1d006af60 msgr2=0x7fa1d006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.356 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.352+0000 7fa1f3036700 1 --2- 172.21.15.91:0/1746145039 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa1d006af60 0x7fa1d006d420 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec06da40 tx=0x7fa1e0008030 comp rx=0 tx=0).stop 2023-11-24T06:09:31.356 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.352+0000 7fa1f3036700 1 -- 172.21.15.91:0/1746145039 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa1ec0b5ab0 msgr2=0x7fa1ec141fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.356 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.352+0000 7fa1f3036700 1 --2- 172.21.15.91:0/1746145039 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa1ec0b5ab0 0x7fa1ec141fc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800c520 tx=0x7fa1d800c830 comp rx=0 tx=0).stop 2023-11-24T06:09:31.356 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.352+0000 7fa1f3036700 1 -- 172.21.15.91:0/1746145039 shutdown_connections 2023-11-24T06:09:31.356 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.352+0000 7fa1f3036700 1 --2- 172.21.15.91:0/1746145039 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa1d006af60 0x7fa1d006d420 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.356 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.352+0000 7fa1f3036700 1 --2- 172.21.15.91:0/1746145039 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa1ec0c70c0 0x7fa1ec1e7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.357 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.352+0000 7fa1f3036700 1 --2- 172.21.15.91:0/1746145039 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa1ec06ee60 0x7fa1ec0c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.357 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.352+0000 7fa1f3036700 1 --2- 172.21.15.91:0/1746145039 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa1ec0b5ab0 0x7fa1ec141fc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.357 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.352+0000 7fa1f3036700 1 -- 172.21.15.91:0/1746145039 >> 172.21.15.91:0/1746145039 conn(0x7fa1ec0b2740 msgr2=0x7fa1ec0b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:31.357 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.352+0000 7fa1f3036700 1 -- 172.21.15.91:0/1746145039 shutdown_connections 2023-11-24T06:09:31.357 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.352+0000 7fa1f3036700 1 -- 172.21.15.91:0/1746145039 wait complete. 2023-11-24T06:09:31.357 INFO:teuthology.orchestra.run.smithi091.stderr:instructed osd(s) 5 to deep-scrub 2023-11-24T06:09:31.367 INFO:tasks.ceph:Scrubbing osd.6 2023-11-24T06:09:31.367 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-11-24T06:09:31.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.488+0000 7f1bdab85700 1 Processor -- start 2023-11-24T06:09:31.492 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.488+0000 7f1bdab85700 1 -- start start 2023-11-24T06:09:31.494 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.489+0000 7f1bdab85700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1bd415e010 0x7f1bd40b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.489+0000 7f1bdab85700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f1bd406caf0 0x7f1bd40ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.489+0000 7f1bdab85700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd40cc2a0 con 0x7f1bd40cea10 2023-11-24T06:09:31.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.489+0000 7f1bdab85700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bd40cee80 con 0x7f1bd406caf0 2023-11-24T06:09:31.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.489+0000 7f1bdab85700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd406d2d0 con 0x7f1bd415e010 2023-11-24T06:09:31.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.489+0000 7f1bd3fff700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f1bd406caf0 0x7f1bd40ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.489+0000 7f1bd9122700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f1bd40cea10 0x7f1bd40b6d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55498/0 (socket says 172.21.15.91:55498) 2023-11-24T06:09:31.495 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.489+0000 7f1bd9122700 1 -- 172.21.15.91:0/560719402 learned_addr learned my addr 172.21.15.91:0/560719402 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:31.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.489+0000 7f1bd3fff700 1 --2- 172.21.15.91:0/560719402 >> v2:172.21.15.91:3300/0 conn(0x7f1bd406caf0 0x7f1bd40ce4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60562/0 (socket says 172.21.15.91:60562) 2023-11-24T06:09:31.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.489+0000 7f1bd8921700 1 --2- 172.21.15.91:0/560719402 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1bd415e010 0x7f1bd40b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.489+0000 7f1bd3fff700 1 -- 172.21.15.91:0/560719402 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1bd415e010 msgr2=0x7f1bd40b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:31.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.489+0000 7f1bd3fff700 1 --2- 172.21.15.91:0/560719402 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1bd415e010 0x7f1bd40b4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.489+0000 7f1bd3fff700 1 -- 172.21.15.91:0/560719402 >> v1:172.21.15.91:6789/0 conn(0x7f1bd40cea10 legacy=0x7f1bd40b6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.489+0000 7f1bd3fff700 1 -- 172.21.15.91:0/560719402 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd40cc560 con 0x7f1bd406caf0 2023-11-24T06:09:31.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.489+0000 7f1bd37fe700 1 -- 172.21.15.91:0/560719402 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928176631 0 0) 0x7f1bd40cc2a0 con 0x7f1bd40cea10 2023-11-24T06:09:31.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.489+0000 7f1bd8921700 1 --2- 172.21.15.91:0/560719402 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1bd415e010 0x7f1bd40b4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:31.496 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.490+0000 7f1bd3fff700 1 --2- 172.21.15.91:0/560719402 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1bd406caf0 0x7f1bd40ce4d0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0009840 tx=0x7f1bc0009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3249dff7d6f75c92 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:31.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.490+0000 7f1bd37fe700 1 -- 172.21.15.91:0/560719402 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0011830 con 0x7f1bd406caf0 2023-11-24T06:09:31.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.490+0000 7f1bd37fe700 1 -- 172.21.15.91:0/560719402 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc0011990 con 0x7f1bd406caf0 2023-11-24T06:09:31.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.490+0000 7f1bd37fe700 1 -- 172.21.15.91:0/560719402 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0011d20 con 0x7f1bd406caf0 2023-11-24T06:09:31.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.490+0000 7f1bdab85700 1 -- 172.21.15.91:0/560719402 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1bd406caf0 msgr2=0x7f1bd40ce4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.490+0000 7f1bdab85700 1 --2- 172.21.15.91:0/560719402 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1bd406caf0 0x7f1bd40ce4d0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0009840 tx=0x7f1bc0009b50 comp rx=0 tx=0).stop 2023-11-24T06:09:31.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.490+0000 7f1bdab85700 1 -- 172.21.15.91:0/560719402 shutdown_connections 2023-11-24T06:09:31.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.490+0000 7f1bdab85700 1 --2- 172.21.15.91:0/560719402 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1bd415e010 0x7f1bd40b4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.497 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.490+0000 7f1bdab85700 1 --2- 172.21.15.91:0/560719402 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1bd406caf0 0x7f1bd40ce4d0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.498 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.490+0000 7f1bdab85700 1 -- 172.21.15.91:0/560719402 >> 172.21.15.91:0/560719402 conn(0x7f1bd40b3800 msgr2=0x7f1bd40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:31.498 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.490+0000 7f1bdab85700 1 -- 172.21.15.91:0/560719402 shutdown_connections 2023-11-24T06:09:31.498 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.490+0000 7f1bdab85700 1 -- 172.21.15.91:0/560719402 wait complete. 2023-11-24T06:09:31.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.492+0000 7f1bdab85700 1 Processor -- start 2023-11-24T06:09:31.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.492+0000 7f1bdab85700 1 -- start start 2023-11-24T06:09:31.501 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.493+0000 7f1bdab85700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1bd406caf0 0x7f1bd41507f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.493+0000 7f1bdab85700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bd40cea10 0x7f1bd4150f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.493+0000 7f1bdab85700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1bd415e010 0x7f1bd4160ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.493+0000 7f1bdab85700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd40ca100 con 0x7f1bd415e010 2023-11-24T06:09:31.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.493+0000 7f1bdab85700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd4150d30 con 0x7f1bd40cea10 2023-11-24T06:09:31.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.493+0000 7f1bdab85700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd40ca3a0 con 0x7f1bd406caf0 2023-11-24T06:09:31.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.493+0000 7f1bd9122700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1bd415e010 0x7f1bd4160ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.502 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.493+0000 7f1bd9122700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1bd415e010 0x7f1bd4160ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60570/0 (socket says 172.21.15.91:60570) 2023-11-24T06:09:31.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.493+0000 7f1bd9122700 1 -- 172.21.15.91:0/3502953319 learned_addr learned my addr 172.21.15.91:0/3502953319 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:31.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.493+0000 7f1bd8921700 1 --2- 172.21.15.91:0/3502953319 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1bd406caf0 0x7f1bd41507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.493+0000 7f1bd3fff700 1 --2- 172.21.15.91:0/3502953319 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bd40cea10 0x7f1bd4150f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.493+0000 7f1bd9122700 1 -- 172.21.15.91:0/3502953319 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1bd406caf0 msgr2=0x7f1bd41507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.493+0000 7f1bd9122700 1 --2- 172.21.15.91:0/3502953319 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f1bd406caf0 0x7f1bd41507f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.493+0000 7f1bd9122700 1 -- 172.21.15.91:0/3502953319 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bd40cea10 msgr2=0x7f1bd4150f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.493+0000 7f1bd9122700 1 --2- 172.21.15.91:0/3502953319 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f1bd40cea10 0x7f1bd4150f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.503 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.493+0000 7f1bd9122700 1 -- 172.21.15.91:0/3502953319 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc00094a0 con 0x7f1bd415e010 2023-11-24T06:09:31.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.494+0000 7f1bd9122700 1 --2- 172.21.15.91:0/3502953319 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1bd415e010 0x7f1bd4160ed0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00cca0 tx=0x7f1bcc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:31.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.494+0000 7f1bd17fa700 1 -- 172.21.15.91:0/3502953319 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc00ab70 con 0x7f1bd415e010 2023-11-24T06:09:31.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.494+0000 7f1bd17fa700 1 -- 172.21.15.91:0/3502953319 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bcc00acd0 con 0x7f1bd415e010 2023-11-24T06:09:31.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.494+0000 7f1bdab85700 1 -- 172.21.15.91:0/3502953319 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd40ca680 con 0x7f1bd415e010 2023-11-24T06:09:31.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.494+0000 7f1bd17fa700 1 -- 172.21.15.91:0/3502953319 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc007310 con 0x7f1bd415e010 2023-11-24T06:09:31.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.494+0000 7f1bdab85700 1 -- 172.21.15.91:0/3502953319 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd40cac40 con 0x7f1bd415e010 2023-11-24T06:09:31.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.495+0000 7f1bdab85700 1 -- 172.21.15.91:0/3502953319 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1bd40b4b00 con 0x7f1bd415e010 2023-11-24T06:09:31.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.495+0000 7f1bd17fa700 1 -- 172.21.15.91:0/3502953319 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f1bcc007470 con 0x7f1bd415e010 2023-11-24T06:09:31.504 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.496+0000 7f1bd17fa700 1 --2- 172.21.15.91:0/3502953319 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1bb806af60 0x7f1bb806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.505 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.496+0000 7f1bd17fa700 1 -- 172.21.15.91:0/3502953319 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1bcc089660 con 0x7f1bd415e010 2023-11-24T06:09:31.505 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.496+0000 7f1bd8921700 1 --2- 172.21.15.91:0/3502953319 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1bb806af60 0x7f1bb806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.505 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.496+0000 7f1bd17fa700 1 --2- 172.21.15.91:0/3502953319 >> [v2:172.21.15.94:6800/4250977597,v1:172.21.15.94:6801/4250977597] conn(0x7f1bb8070dc0 0x7f1bb80731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.505 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.496+0000 7f1bd17fa700 1 -- 172.21.15.91:0/3502953319 --> [v2:172.21.15.94:6800/4250977597,v1:172.21.15.94:6801/4250977597] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1bb80738b0 con 0x7f1bb8070dc0 2023-11-24T06:09:31.505 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.496+0000 7f1bd17fa700 1 -- 172.21.15.91:0/3502953319 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bcc089b00 con 0x7f1bd415e010 2023-11-24T06:09:31.505 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.496+0000 7f1bd3fff700 1 --2- 172.21.15.91:0/3502953319 >> [v2:172.21.15.94:6800/4250977597,v1:172.21.15.94:6801/4250977597] conn(0x7f1bb8070dc0 0x7f1bb80731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.505 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.496+0000 7f1bd8921700 1 --2- 172.21.15.91:0/3502953319 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1bb806af60 0x7f1bb806d420 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8006fd0 tx=0x7f1bc8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:31.506 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.496+0000 7f1bd3fff700 1 --2- 172.21.15.91:0/3502953319 >> [v2:172.21.15.94:6800/4250977597,v1:172.21.15.94:6801/4250977597] conn(0x7f1bb8070dc0 0x7f1bb80731e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:31.506 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.497+0000 7f1bd17fa700 1 -- 172.21.15.91:0/3502953319 <== osd.6 v2:172.21.15.94:6800/4250977597 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1bb80738b0 con 0x7f1bb8070dc0 2023-11-24T06:09:31.515 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.512+0000 7f1bdab85700 1 -- 172.21.15.91:0/3502953319 --> [v2:172.21.15.94:6800/4250977597,v1:172.21.15.94:6801/4250977597] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1bd40ccb20 con 0x7f1bb8070dc0 2023-11-24T06:09:31.516 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.513+0000 7f1bd17fa700 1 -- 172.21.15.91:0/3502953319 <== osd.6 v2:172.21.15.94:6800/4250977597 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f1bc0022040 con 0x7f1bb8070dc0 2023-11-24T06:09:31.516 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2023-11-24T06:09:31.517 INFO:teuthology.orchestra.run.smithi091.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T06:09:31.517 INFO:teuthology.orchestra.run.smithi091.stdout:} 2023-11-24T06:09:31.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.513+0000 7f1bdab85700 1 -- 172.21.15.91:0/3502953319 >> [v2:172.21.15.94:6800/4250977597,v1:172.21.15.94:6801/4250977597] conn(0x7f1bb8070dc0 msgr2=0x7f1bb80731e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.513+0000 7f1bdab85700 1 --2- 172.21.15.91:0/3502953319 >> [v2:172.21.15.94:6800/4250977597,v1:172.21.15.94:6801/4250977597] conn(0x7f1bb8070dc0 0x7f1bb80731e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.513+0000 7f1bdab85700 1 -- 172.21.15.91:0/3502953319 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1bb806af60 msgr2=0x7f1bb806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.513+0000 7f1bdab85700 1 --2- 172.21.15.91:0/3502953319 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f1bb806af60 0x7f1bb806d420 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8006fd0 tx=0x7f1bc8008040 comp rx=0 tx=0).stop 2023-11-24T06:09:31.518 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.513+0000 7f1bdab85700 1 -- 172.21.15.91:0/3502953319 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1bd415e010 msgr2=0x7f1bd4160ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.513+0000 7f1bdab85700 1 --2- 172.21.15.91:0/3502953319 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f1bd415e010 0x7f1bd4160ed0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00cca0 tx=0x7f1bcc00a060 comp rx=0 tx=0).stop 2023-11-24T06:09:31.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.513+0000 7f1bd9122700 1 -- 172.21.15.91:0/3502953319 reap_dead start 2023-11-24T06:09:31.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.513+0000 7f1bdab85700 1 -- 172.21.15.91:0/3502953319 shutdown_connections 2023-11-24T06:09:31.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.513+0000 7f1bdab85700 1 -- 172.21.15.91:0/3502953319 >> 172.21.15.91:0/3502953319 conn(0x7f1bd40b3800 msgr2=0x7f1bd40d16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:31.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.513+0000 7f1bdab85700 1 -- 172.21.15.91:0/3502953319 shutdown_connections 2023-11-24T06:09:31.519 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.513+0000 7f1bdab85700 1 -- 172.21.15.91:0/3502953319 wait complete. 2023-11-24T06:09:31.528 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-11-24T06:09:31.649 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.646+0000 7fa7994cf700 1 Processor -- start 2023-11-24T06:09:31.650 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.646+0000 7fa7994cf700 1 -- start start 2023-11-24T06:09:31.652 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.646+0000 7fa7994cf700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fa7941445f0 0x7fa7940cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.652 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.646+0000 7fa7994cf700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fa79406cae0 0x7fa7940726f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.652 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.646+0000 7fa7994cf700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa79406e480 0x7fa79406e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.646+0000 7fa7994cf700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa79406ee50 con 0x7fa7941445f0 2023-11-24T06:09:31.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.646+0000 7fa7994cf700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7941449d0 con 0x7fa79406cae0 2023-11-24T06:09:31.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.646+0000 7fa7994cf700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa794071650 con 0x7fa79406e480 2023-11-24T06:09:31.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.647+0000 7fa792ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fa7941445f0 0x7fa7940cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.647+0000 7fa792ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7fa7941445f0 0x7fa7940cb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60576/0 (socket says 172.21.15.91:60576) 2023-11-24T06:09:31.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.647+0000 7fa792ffd700 1 -- 172.21.15.91:0/3072497546 learned_addr learned my addr 172.21.15.91:0/3072497546 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:31.653 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.647+0000 7fa7937fe700 1 --2- 172.21.15.91:0/3072497546 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa79406e480 0x7fa79406e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.647+0000 7fa7927fc700 1 --2- 172.21.15.91:0/3072497546 >> v2:172.21.15.94:3300/0 conn(0x7fa79406cae0 0x7fa7940726f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.647+0000 7fa792ffd700 1 -- 172.21.15.91:0/3072497546 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa79406e480 msgr2=0x7fa79406e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:31.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.647+0000 7fa792ffd700 1 --2- 172.21.15.91:0/3072497546 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa79406e480 0x7fa79406e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.647+0000 7fa792ffd700 1 -- 172.21.15.91:0/3072497546 >> v2:172.21.15.94:3300/0 conn(0x7fa79406cae0 msgr2=0x7fa7940726f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:31.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.647+0000 7fa792ffd700 1 --2- 172.21.15.91:0/3072497546 >> v2:172.21.15.94:3300/0 conn(0x7fa79406cae0 0x7fa7940726f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.647+0000 7fa792ffd700 1 -- 172.21.15.91:0/3072497546 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa794071930 con 0x7fa7941445f0 2023-11-24T06:09:31.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.647+0000 7fa7927fc700 1 --2- 172.21.15.91:0/3072497546 >> v2:172.21.15.94:3300/0 conn(0x7fa79406cae0 0x7fa7940726f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:31.654 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.647+0000 7fa792ffd700 1 --2- 172.21.15.91:0/3072497546 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa7941445f0 0x7fa7940cb2f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa7940af590 tx=0x7fa77c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45699ddf9c524ee7 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:31.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.647+0000 7fa791ffb700 1 -- 172.21.15.91:0/3072497546 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77c011830 con 0x7fa7941445f0 2023-11-24T06:09:31.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.647+0000 7fa791ffb700 1 -- 172.21.15.91:0/3072497546 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77c011990 con 0x7fa7941445f0 2023-11-24T06:09:31.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.647+0000 7fa791ffb700 1 -- 172.21.15.91:0/3072497546 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77c011d20 con 0x7fa7941445f0 2023-11-24T06:09:31.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.648+0000 7fa7994cf700 1 -- 172.21.15.91:0/3072497546 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa7941445f0 msgr2=0x7fa7940cb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.648+0000 7fa7994cf700 1 --2- 172.21.15.91:0/3072497546 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa7941445f0 0x7fa7940cb2f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa7940af590 tx=0x7fa77c009bd0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.648+0000 7fa7994cf700 1 -- 172.21.15.91:0/3072497546 shutdown_connections 2023-11-24T06:09:31.655 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.648+0000 7fa7994cf700 1 --2- 172.21.15.91:0/3072497546 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa79406e480 0x7fa79406e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.648+0000 7fa7994cf700 1 --2- 172.21.15.91:0/3072497546 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa7941445f0 0x7fa7940cb2f0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.648+0000 7fa7994cf700 1 --2- 172.21.15.91:0/3072497546 >> v2:172.21.15.94:3300/0 conn(0x7fa79406cae0 0x7fa7940726f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.648+0000 7fa7994cf700 1 -- 172.21.15.91:0/3072497546 >> 172.21.15.91:0/3072497546 conn(0x7fa7940cf980 msgr2=0x7fa7940cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:31.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.648+0000 7fa7994cf700 1 -- 172.21.15.91:0/3072497546 shutdown_connections 2023-11-24T06:09:31.656 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.648+0000 7fa7994cf700 1 -- 172.21.15.91:0/3072497546 wait complete. 2023-11-24T06:09:31.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.649+0000 7fa7994cf700 1 Processor -- start 2023-11-24T06:09:31.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.649+0000 7fa7994cf700 1 -- start start 2023-11-24T06:09:31.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa7994cf700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa79406cae0 0x7fa7941577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa7994cf700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa79406e480 0x7fa794140340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa7994cf700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa7941445f0 0x7fa794140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa7994cf700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79415a0c0 con 0x7fa7941445f0 2023-11-24T06:09:31.659 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa7994cf700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7941f15f0 con 0x7fa79406e480 2023-11-24T06:09:31.660 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa7994cf700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa794140f00 con 0x7fa79406cae0 2023-11-24T06:09:31.660 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa7937fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa7941445f0 0x7fa794140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.660 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa7937fe700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa7941445f0 0x7fa794140880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60586/0 (socket says 172.21.15.91:60586) 2023-11-24T06:09:31.660 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa7937fe700 1 -- 172.21.15.91:0/1835452336 learned_addr learned my addr 172.21.15.91:0/1835452336 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:31.660 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa7927fc700 1 --2- 172.21.15.91:0/1835452336 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa79406e480 0x7fa794140340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.660 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa792ffd700 1 --2- 172.21.15.91:0/1835452336 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa79406cae0 0x7fa7941577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.660 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa7937fe700 1 -- 172.21.15.91:0/1835452336 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa79406cae0 msgr2=0x7fa7941577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa7937fe700 1 --2- 172.21.15.91:0/1835452336 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa79406cae0 0x7fa7941577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa7937fe700 1 -- 172.21.15.91:0/1835452336 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa79406e480 msgr2=0x7fa794140340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa7937fe700 1 --2- 172.21.15.91:0/1835452336 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa79406e480 0x7fa794140340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa7937fe700 1 -- 172.21.15.91:0/1835452336 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c009550 con 0x7fa7941445f0 2023-11-24T06:09:31.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa792ffd700 1 --2- 172.21.15.91:0/1835452336 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa79406cae0 0x7fa7941577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:31.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa7937fe700 1 --2- 172.21.15.91:0/1835452336 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa7941445f0 0x7fa794140880 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa78c00efc0 tx=0x7fa78c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:31.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.650+0000 7fa78b7fe700 1 -- 172.21.15.91:0/1835452336 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c00cd40 con 0x7fa7941445f0 2023-11-24T06:09:31.661 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.651+0000 7fa78b7fe700 1 -- 172.21.15.91:0/1835452336 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78c00f040 con 0x7fa7941445f0 2023-11-24T06:09:31.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.651+0000 7fa7994cf700 1 -- 172.21.15.91:0/1835452336 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7941411e0 con 0x7fa7941445f0 2023-11-24T06:09:31.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.651+0000 7fa78b7fe700 1 -- 172.21.15.91:0/1835452336 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c0146c0 con 0x7fa7941445f0 2023-11-24T06:09:31.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.651+0000 7fa7994cf700 1 -- 172.21.15.91:0/1835452336 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7941417a0 con 0x7fa7941445f0 2023-11-24T06:09:31.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.651+0000 7fa7994cf700 1 -- 172.21.15.91:0/1835452336 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7941f1810 con 0x7fa7941445f0 2023-11-24T06:09:31.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.652+0000 7fa78b7fe700 1 -- 172.21.15.91:0/1835452336 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fa78c014820 con 0x7fa7941445f0 2023-11-24T06:09:31.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.652+0000 7fa78b7fe700 1 --2- 172.21.15.91:0/1835452336 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa77406af60 0x7fa77406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:31.662 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.652+0000 7fa78b7fe700 1 -- 172.21.15.91:0/1835452336 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa78c089bf0 con 0x7fa7941445f0 2023-11-24T06:09:31.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.652+0000 7fa792ffd700 1 --2- 172.21.15.91:0/1835452336 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa77406af60 0x7fa77406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:31.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.653+0000 7fa792ffd700 1 --2- 172.21.15.91:0/1835452336 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa77406af60 0x7fa77406d420 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa77c009fd0 tx=0x7fa77c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:31.663 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.657+0000 7fa78b7fe700 1 -- 172.21.15.91:0/1835452336 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa7941f1810 con 0x7fa7941445f0 2023-11-24T06:09:31.800 DEBUG:teuthology.orchestra.run.smithi091:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:09:31.803 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:09:31.859 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.856+0000 7fa7994cf700 1 -- 172.21.15.91:0/1835452336 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7940721d0 con 0x7fa77406af60 2023-11-24T06:09:31.860 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.856+0000 7fa78b7fe700 1 -- 172.21.15.91:0/1835452336 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa77c011b00 con 0x7fa77406af60 2023-11-24T06:09:31.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.858+0000 7fa7994cf700 1 -- 172.21.15.91:0/1835452336 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa77406af60 msgr2=0x7fa77406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.858+0000 7fa7994cf700 1 --2- 172.21.15.91:0/1835452336 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa77406af60 0x7fa77406d420 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa77c009fd0 tx=0x7fa77c008030 comp rx=0 tx=0).stop 2023-11-24T06:09:31.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.858+0000 7fa7994cf700 1 -- 172.21.15.91:0/1835452336 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa7941445f0 msgr2=0x7fa794140880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:31.862 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.858+0000 7fa7994cf700 1 --2- 172.21.15.91:0/1835452336 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa7941445f0 0x7fa794140880 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa78c00efc0 tx=0x7fa78c00c3a0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.858+0000 7fa7994cf700 1 -- 172.21.15.91:0/1835452336 shutdown_connections 2023-11-24T06:09:31.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.858+0000 7fa7994cf700 1 --2- 172.21.15.91:0/1835452336 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fa77406af60 0x7fa77406d420 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.858+0000 7fa7994cf700 1 --2- 172.21.15.91:0/1835452336 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fa7941445f0 0x7fa794140880 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.858+0000 7fa7994cf700 1 --2- 172.21.15.91:0/1835452336 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fa79406cae0 0x7fa7941577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.858+0000 7fa7994cf700 1 --2- 172.21.15.91:0/1835452336 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fa79406e480 0x7fa794140340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:31.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.858+0000 7fa7994cf700 1 -- 172.21.15.91:0/1835452336 >> 172.21.15.91:0/1835452336 conn(0x7fa7940cf980 msgr2=0x7fa7940ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:31.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.858+0000 7fa7994cf700 1 -- 172.21.15.91:0/1835452336 shutdown_connections 2023-11-24T06:09:31.863 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:31.858+0000 7fa7994cf700 1 -- 172.21.15.91:0/1835452336 wait complete. 2023-11-24T06:09:31.864 INFO:teuthology.orchestra.run.smithi091.stderr:instructed osd(s) 6 to deep-scrub 2023-11-24T06:09:31.874 INFO:tasks.ceph:Scrubbing osd.7 2023-11-24T06:09:31.874 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-11-24T06:09:32.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.035+0000 7ff5ccae1700 1 Processor -- start 2023-11-24T06:09:32.039 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.035+0000 7ff5ccae1700 1 -- start start 2023-11-24T06:09:32.041 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.036+0000 7ff5ccae1700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff5c806afd0 0x7ff5c80b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.042 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.036+0000 7ff5ccae1700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff5c8072350 0x7ff5c80727a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.042 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.036+0000 7ff5ccae1700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c8071ec0 con 0x7ff5c806b3b0 2023-11-24T06:09:32.042 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.036+0000 7ff5ccae1700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5c8071b90 con 0x7ff5c806afd0 2023-11-24T06:09:32.042 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.036+0000 7ff5ccae1700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c815ed30 con 0x7ff5c8072350 2023-11-24T06:09:32.042 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.037+0000 7ff5c659c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff5c806afd0 0x7ff5c80b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.042 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.037+0000 7ff5c659c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff5c806afd0 0x7ff5c80b59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60590/0 (socket says 172.21.15.91:60590) 2023-11-24T06:09:32.042 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.037+0000 7ff5c659c700 1 -- 172.21.15.91:0/69644834 learned_addr learned my addr 172.21.15.91:0/69644834 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:32.042 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.037+0000 7ff5c659c700 1 -- 172.21.15.91:0/69644834 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff5c8072350 msgr2=0x7ff5c80727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:32.043 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.037+0000 7ff5c659c700 1 --2- 172.21.15.91:0/69644834 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff5c8072350 0x7ff5c80727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.043 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.037+0000 7ff5c659c700 1 -- 172.21.15.91:0/69644834 >> v1:172.21.15.94:6789/0 conn(0x7ff5c806b3b0 legacy=0x7ff5c815e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:32.043 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.037+0000 7ff5c659c700 1 -- 172.21.15.91:0/69644834 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c815f010 con 0x7ff5c806afd0 2023-11-24T06:09:32.043 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.037+0000 7ff5c659c700 1 --2- 172.21.15.91:0/69644834 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff5c806afd0 0x7ff5c80b59d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0098f0 tx=0x7ff5bc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828012dbc2ae175a server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:32.043 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.037+0000 7ff5c559a700 1 -- 172.21.15.91:0/69644834 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc004230 con 0x7ff5c806afd0 2023-11-24T06:09:32.043 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.037+0000 7ff5c559a700 1 -- 172.21.15.91:0/69644834 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5bc004390 con 0x7ff5c806afd0 2023-11-24T06:09:32.043 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.037+0000 7ff5c559a700 1 -- 172.21.15.91:0/69644834 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc0036a0 con 0x7ff5c806afd0 2023-11-24T06:09:32.043 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.038+0000 7ff5ccae1700 1 -- 172.21.15.91:0/69644834 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff5c806afd0 msgr2=0x7ff5c80b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.044 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.038+0000 7ff5ccae1700 1 --2- 172.21.15.91:0/69644834 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff5c806afd0 0x7ff5c80b59d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0098f0 tx=0x7ff5bc009c00 comp rx=0 tx=0).stop 2023-11-24T06:09:32.044 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.038+0000 7ff5ccae1700 1 -- 172.21.15.91:0/69644834 shutdown_connections 2023-11-24T06:09:32.044 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.038+0000 7ff5ccae1700 1 --2- 172.21.15.91:0/69644834 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff5c8072350 0x7ff5c80727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.044 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.038+0000 7ff5ccae1700 1 --2- 172.21.15.91:0/69644834 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff5c806afd0 0x7ff5c80b59d0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.044 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.038+0000 7ff5ccae1700 1 -- 172.21.15.91:0/69644834 >> 172.21.15.91:0/69644834 conn(0x7ff5c80b2760 msgr2=0x7ff5c80b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:32.044 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.038+0000 7ff5ccae1700 1 -- 172.21.15.91:0/69644834 shutdown_connections 2023-11-24T06:09:32.044 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.038+0000 7ff5ccae1700 1 -- 172.21.15.91:0/69644834 wait complete. 2023-11-24T06:09:32.047 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.039+0000 7ff5ccae1700 1 Processor -- start 2023-11-24T06:09:32.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.039+0000 7ff5ccae1700 1 -- start start 2023-11-24T06:09:32.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5ccae1700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff5c806afd0 0x7ff5c813dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5ccae1700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff5c806b3b0 0x7ff5c81465e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5ccae1700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5c8072350 0x7ff5c815c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5ccae1700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c81402f0 con 0x7ff5c806b3b0 2023-11-24T06:09:32.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5ccae1700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c81405d0 con 0x7ff5c8072350 2023-11-24T06:09:32.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5ccae1700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c81408b0 con 0x7ff5c806afd0 2023-11-24T06:09:32.048 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5c5d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff5c806b3b0 0x7ff5c81465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.049 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5c5d9b700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff5c806b3b0 0x7ff5c81465e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60598/0 (socket says 172.21.15.91:60598) 2023-11-24T06:09:32.049 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5c5d9b700 1 -- 172.21.15.91:0/882444879 learned_addr learned my addr 172.21.15.91:0/882444879 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:32.049 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5c659c700 1 --2- 172.21.15.91:0/882444879 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff5c806afd0 0x7ff5c813dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.049 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5c6d9d700 1 --2- 172.21.15.91:0/882444879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5c8072350 0x7ff5c815c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.049 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5c5d9b700 1 -- 172.21.15.91:0/882444879 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff5c806afd0 msgr2=0x7ff5c813dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.049 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5c5d9b700 1 --2- 172.21.15.91:0/882444879 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff5c806afd0 0x7ff5c813dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.049 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5c5d9b700 1 -- 172.21.15.91:0/882444879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5c8072350 msgr2=0x7ff5c815c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.049 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5c5d9b700 1 --2- 172.21.15.91:0/882444879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5c8072350 0x7ff5c815c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.050 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5c5d9b700 1 -- 172.21.15.91:0/882444879 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5bc009550 con 0x7ff5c806b3b0 2023-11-24T06:09:32.050 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5c6d9d700 1 --2- 172.21.15.91:0/882444879 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff5c8072350 0x7ff5c815c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:32.050 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5c5d9b700 1 --2- 172.21.15.91:0/882444879 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff5c806b3b0 0x7ff5c81465e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff5b800c4a0 tx=0x7ff5b800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:32.050 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.040+0000 7ff5b6ffd700 1 -- 172.21.15.91:0/882444879 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b80054e0 con 0x7ff5c806b3b0 2023-11-24T06:09:32.050 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.041+0000 7ff5b6ffd700 1 -- 172.21.15.91:0/882444879 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b8003680 con 0x7ff5c806b3b0 2023-11-24T06:09:32.050 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.041+0000 7ff5ccae1700 1 -- 172.21.15.91:0/882444879 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c8140b90 con 0x7ff5c806b3b0 2023-11-24T06:09:32.051 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.041+0000 7ff5b6ffd700 1 -- 172.21.15.91:0/882444879 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b8003a30 con 0x7ff5c806b3b0 2023-11-24T06:09:32.051 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.041+0000 7ff5ccae1700 1 -- 172.21.15.91:0/882444879 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c8146b20 con 0x7ff5c806b3b0 2023-11-24T06:09:32.052 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.041+0000 7ff5ccae1700 1 -- 172.21.15.91:0/882444879 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff5c806b840 con 0x7ff5c806b3b0 2023-11-24T06:09:32.052 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.042+0000 7ff5b6ffd700 1 -- 172.21.15.91:0/882444879 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7ff5b8003b90 con 0x7ff5c806b3b0 2023-11-24T06:09:32.052 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.042+0000 7ff5b6ffd700 1 --2- 172.21.15.91:0/882444879 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff5a806af60 0x7ff5a806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.052 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.042+0000 7ff5b6ffd700 1 -- 172.21.15.91:0/882444879 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff5b8088870 con 0x7ff5c806b3b0 2023-11-24T06:09:32.052 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.042+0000 7ff5c659c700 1 --2- 172.21.15.91:0/882444879 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff5a806af60 0x7ff5a806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.052 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.042+0000 7ff5b6ffd700 1 --2- 172.21.15.91:0/882444879 >> [v2:172.21.15.94:6816/3673570051,v1:172.21.15.94:6817/3673570051] conn(0x7ff5a8070dc0 0x7ff5a80731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.053 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.042+0000 7ff5b6ffd700 1 -- 172.21.15.91:0/882444879 --> [v2:172.21.15.94:6816/3673570051,v1:172.21.15.94:6817/3673570051] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff5a80738b0 con 0x7ff5a8070dc0 2023-11-24T06:09:32.053 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.042+0000 7ff5b6ffd700 1 -- 172.21.15.91:0/882444879 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5b808d2c0 con 0x7ff5c806b3b0 2023-11-24T06:09:32.053 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.043+0000 7ff5c6d9d700 1 --2- 172.21.15.91:0/882444879 >> [v2:172.21.15.94:6816/3673570051,v1:172.21.15.94:6817/3673570051] conn(0x7ff5a8070dc0 0x7ff5a80731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.053 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.043+0000 7ff5c659c700 1 --2- 172.21.15.91:0/882444879 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff5a806af60 0x7ff5a806d420 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009130 tx=0x7ff5bc008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:32.053 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.043+0000 7ff5c6d9d700 1 --2- 172.21.15.91:0/882444879 >> [v2:172.21.15.94:6816/3673570051,v1:172.21.15.94:6817/3673570051] conn(0x7ff5a8070dc0 0x7ff5a80731e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:32.053 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.044+0000 7ff5b6ffd700 1 -- 172.21.15.91:0/882444879 <== osd.7 v2:172.21.15.94:6816/3673570051 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff5a80738b0 con 0x7ff5a8070dc0 2023-11-24T06:09:32.062 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.059+0000 7ff5ccae1700 1 -- 172.21.15.91:0/882444879 --> [v2:172.21.15.94:6816/3673570051,v1:172.21.15.94:6817/3673570051] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff5c8146cb0 con 0x7ff5a8070dc0 2023-11-24T06:09:32.063 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.059+0000 7ff5b6ffd700 1 -- 172.21.15.91:0/882444879 <== osd.7 v2:172.21.15.94:6816/3673570051 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff5c8146cb0 con 0x7ff5a8070dc0 2023-11-24T06:09:32.063 INFO:teuthology.orchestra.run.smithi091.stdout:{ 2023-11-24T06:09:32.063 INFO:teuthology.orchestra.run.smithi091.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-11-24T06:09:32.063 INFO:teuthology.orchestra.run.smithi091.stdout:} 2023-11-24T06:09:32.064 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.060+0000 7ff5ccae1700 1 -- 172.21.15.91:0/882444879 >> [v2:172.21.15.94:6816/3673570051,v1:172.21.15.94:6817/3673570051] conn(0x7ff5a8070dc0 msgr2=0x7ff5a80731e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.065 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.060+0000 7ff5ccae1700 1 --2- 172.21.15.91:0/882444879 >> [v2:172.21.15.94:6816/3673570051,v1:172.21.15.94:6817/3673570051] conn(0x7ff5a8070dc0 0x7ff5a80731e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.065 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.060+0000 7ff5ccae1700 1 -- 172.21.15.91:0/882444879 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff5a806af60 msgr2=0x7ff5a806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.065 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.060+0000 7ff5ccae1700 1 --2- 172.21.15.91:0/882444879 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff5a806af60 0x7ff5a806d420 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009130 tx=0x7ff5bc008030 comp rx=0 tx=0).stop 2023-11-24T06:09:32.065 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.060+0000 7ff5ccae1700 1 -- 172.21.15.91:0/882444879 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff5c806b3b0 msgr2=0x7ff5c81465e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.065 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.060+0000 7ff5ccae1700 1 --2- 172.21.15.91:0/882444879 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff5c806b3b0 0x7ff5c81465e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff5b800c4a0 tx=0x7ff5b800c860 comp rx=0 tx=0).stop 2023-11-24T06:09:32.065 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.060+0000 7ff5c6d9d700 1 -- 172.21.15.91:0/882444879 reap_dead start 2023-11-24T06:09:32.065 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.060+0000 7ff5ccae1700 1 -- 172.21.15.91:0/882444879 shutdown_connections 2023-11-24T06:09:32.065 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.060+0000 7ff5ccae1700 1 -- 172.21.15.91:0/882444879 >> 172.21.15.91:0/882444879 conn(0x7ff5c80b2760 msgr2=0x7ff5c81426a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:32.066 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.060+0000 7ff5ccae1700 1 -- 172.21.15.91:0/882444879 shutdown_connections 2023-11-24T06:09:32.066 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.060+0000 7ff5ccae1700 1 -- 172.21.15.91:0/882444879 wait complete. 2023-11-24T06:09:32.075 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-11-24T06:09:32.196 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.192+0000 7f482084f700 1 Processor -- start 2023-11-24T06:09:32.196 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.192+0000 7f482084f700 1 -- start start 2023-11-24T06:09:32.197 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.193+0000 7f482084f700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f481c0b4150 0x7f481c06de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.193+0000 7f482084f700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7f481c0b5ab0 0x7f481c06e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.193+0000 7f482084f700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c071f50 con 0x7f481c06b620 2023-11-24T06:09:32.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.193+0000 7f482084f700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f481c072230 con 0x7f481c0b4150 2023-11-24T06:09:32.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.193+0000 7f482084f700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f481c072510 con 0x7f481c0b5ab0 2023-11-24T06:09:32.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.193+0000 7f4819d9b700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f481c0b4150 0x7f481c06de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.193+0000 7f4819d9b700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f481c0b4150 0x7f481c06de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60608/0 (socket says 172.21.15.91:60608) 2023-11-24T06:09:32.198 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.193+0000 7f4819d9b700 1 -- 172.21.15.91:0/1066416506 learned_addr learned my addr 172.21.15.91:0/1066416506 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:32.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.193+0000 7f481ad9d700 1 --2- 172.21.15.91:0/1066416506 >> v2:172.21.15.94:3300/0 conn(0x7f481c0b5ab0 0x7f481c06e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.193+0000 7f4819d9b700 1 -- 172.21.15.91:0/1066416506 >> v2:172.21.15.94:3300/0 conn(0x7f481c0b5ab0 msgr2=0x7f481c06e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:32.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.193+0000 7f4819d9b700 1 --2- 172.21.15.91:0/1066416506 >> v2:172.21.15.94:3300/0 conn(0x7f481c0b5ab0 0x7f481c06e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.193+0000 7f4819d9b700 1 -- 172.21.15.91:0/1066416506 >> v1:172.21.15.94:6789/0 conn(0x7f481c06b620 legacy=0x7f481c0b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.193+0000 7f4819d9b700 1 -- 172.21.15.91:0/1066416506 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c0727f0 con 0x7f481c0b4150 2023-11-24T06:09:32.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.194+0000 7f4819d9b700 1 --2- 172.21.15.91:0/1066416506 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f481c0b4150 0x7f481c06de50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f481c0694e0 tx=0x7f480c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11fccf51912e0a0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:32.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.194+0000 7f481959a700 1 -- 172.21.15.91:0/1066416506 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480c011830 con 0x7f481c0b4150 2023-11-24T06:09:32.199 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.194+0000 7f481959a700 1 -- 172.21.15.91:0/1066416506 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480c011990 con 0x7f481c0b4150 2023-11-24T06:09:32.200 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.194+0000 7f481959a700 1 -- 172.21.15.91:0/1066416506 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480c011d20 con 0x7f481c0b4150 2023-11-24T06:09:32.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.194+0000 7f482084f700 1 -- 172.21.15.91:0/1066416506 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f481c0b4150 msgr2=0x7f481c06de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.194+0000 7f482084f700 1 --2- 172.21.15.91:0/1066416506 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f481c0b4150 0x7f481c06de50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f481c0694e0 tx=0x7f480c009bd0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.194+0000 7f482084f700 1 -- 172.21.15.91:0/1066416506 shutdown_connections 2023-11-24T06:09:32.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.194+0000 7f482084f700 1 --2- 172.21.15.91:0/1066416506 >> v2:172.21.15.94:3300/0 conn(0x7f481c0b5ab0 0x7f481c06e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.194+0000 7f482084f700 1 --2- 172.21.15.91:0/1066416506 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f481c0b4150 0x7f481c06de50 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.203 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.194+0000 7f482084f700 1 -- 172.21.15.91:0/1066416506 >> 172.21.15.91:0/1066416506 conn(0x7f481c0b2740 msgr2=0x7f481c0b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:32.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.194+0000 7f482084f700 1 -- 172.21.15.91:0/1066416506 shutdown_connections 2023-11-24T06:09:32.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.194+0000 7f482084f700 1 -- 172.21.15.91:0/1066416506 wait complete. 2023-11-24T06:09:32.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.196+0000 7f482084f700 1 Processor -- start 2023-11-24T06:09:32.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.196+0000 7f482084f700 1 -- start start 2023-11-24T06:09:32.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.196+0000 7f482084f700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f481c06b620 0x7f481c1f8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.196+0000 7f482084f700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f481c0b4150 0x7f481c1f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.196+0000 7f482084f700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f481c0b5ab0 0x7f481c0b1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.204 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.196+0000 7f482084f700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c136fe0 con 0x7f481c06b620 2023-11-24T06:09:32.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.196+0000 7f482084f700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c133880 con 0x7f481c0b4150 2023-11-24T06:09:32.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.196+0000 7f482084f700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481c1f7530 con 0x7f481c0b5ab0 2023-11-24T06:09:32.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.197+0000 7f481a59c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f481c06b620 0x7f481c1f8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.197+0000 7f481a59c700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f481c06b620 0x7f481c1f8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60612/0 (socket says 172.21.15.91:60612) 2023-11-24T06:09:32.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.197+0000 7f481a59c700 1 -- 172.21.15.91:0/1835295930 learned_addr learned my addr 172.21.15.91:0/1835295930 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:32.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.197+0000 7f4819d9b700 1 --2- 172.21.15.91:0/1835295930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f481c0b4150 0x7f481c1f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.197+0000 7f481ad9d700 1 --2- 172.21.15.91:0/1835295930 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f481c0b5ab0 0x7f481c0b1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.205 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.197+0000 7f481a59c700 1 -- 172.21.15.91:0/1835295930 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f481c0b5ab0 msgr2=0x7f481c0b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.197+0000 7f481a59c700 1 --2- 172.21.15.91:0/1835295930 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f481c0b5ab0 0x7f481c0b1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.197+0000 7f481a59c700 1 -- 172.21.15.91:0/1835295930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f481c0b4150 msgr2=0x7f481c1f3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.197+0000 7f481a59c700 1 --2- 172.21.15.91:0/1835295930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f481c0b4150 0x7f481c1f3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.197+0000 7f481a59c700 1 -- 172.21.15.91:0/1835295930 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480c009550 con 0x7f481c06b620 2023-11-24T06:09:32.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.197+0000 7f481a59c700 1 --2- 172.21.15.91:0/1835295930 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f481c06b620 0x7f481c1f8010 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f481000c4e0 tx=0x7f481000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:32.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.197+0000 7f480affd700 1 -- 172.21.15.91:0/1835295930 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48100052d0 con 0x7f481c06b620 2023-11-24T06:09:32.206 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.197+0000 7f480affd700 1 -- 172.21.15.91:0/1835295930 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4810005430 con 0x7f481c06b620 2023-11-24T06:09:32.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.197+0000 7f482084f700 1 -- 172.21.15.91:0/1835295930 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481c1f7810 con 0x7f481c06b620 2023-11-24T06:09:32.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.197+0000 7f480affd700 1 -- 172.21.15.91:0/1835295930 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48100038c0 con 0x7f481c06b620 2023-11-24T06:09:32.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.197+0000 7f482084f700 1 -- 172.21.15.91:0/1835295930 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f481c133b60 con 0x7f481c06b620 2023-11-24T06:09:32.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.198+0000 7f482084f700 1 -- 172.21.15.91:0/1835295930 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481c068670 con 0x7f481c06b620 2023-11-24T06:09:32.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.198+0000 7f480affd700 1 -- 172.21.15.91:0/1835295930 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f4810003a20 con 0x7f481c06b620 2023-11-24T06:09:32.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.199+0000 7f480affd700 1 --2- 172.21.15.91:0/1835295930 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f47fc06af60 0x7f47fc06d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.199+0000 7f480affd700 1 -- 172.21.15.91:0/1835295930 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4810088830 con 0x7f481c06b620 2023-11-24T06:09:32.207 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.199+0000 7f4819d9b700 1 --2- 172.21.15.91:0/1835295930 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f47fc06af60 0x7f47fc06d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.208 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.199+0000 7f4819d9b700 1 --2- 172.21.15.91:0/1835295930 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f47fc06af60 0x7f47fc06d420 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f480c009c10 tx=0x7f480c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:32.208 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.203+0000 7f480affd700 1 -- 172.21.15.91:0/1835295930 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4810053bf0 con 0x7f481c06b620 2023-11-24T06:09:32.396 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.392+0000 7f482084f700 1 -- 172.21.15.91:0/1835295930 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f481c06ba00 con 0x7f47fc06af60 2023-11-24T06:09:32.396 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.393+0000 7f480affd700 1 -- 172.21.15.91:0/1835295930 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f480c011b80 con 0x7f47fc06af60 2023-11-24T06:09:32.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.395+0000 7f482084f700 1 -- 172.21.15.91:0/1835295930 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f47fc06af60 msgr2=0x7f47fc06d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.395+0000 7f482084f700 1 --2- 172.21.15.91:0/1835295930 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f47fc06af60 0x7f47fc06d420 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f480c009c10 tx=0x7f480c008030 comp rx=0 tx=0).stop 2023-11-24T06:09:32.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.395+0000 7f482084f700 1 -- 172.21.15.91:0/1835295930 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f481c06b620 msgr2=0x7f481c1f8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.395+0000 7f482084f700 1 --2- 172.21.15.91:0/1835295930 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f481c06b620 0x7f481c1f8010 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f481000c4e0 tx=0x7f481000c8a0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.395+0000 7f482084f700 1 -- 172.21.15.91:0/1835295930 shutdown_connections 2023-11-24T06:09:32.399 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.395+0000 7f482084f700 1 --2- 172.21.15.91:0/1835295930 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f47fc06af60 0x7f47fc06d420 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.395+0000 7f482084f700 1 --2- 172.21.15.91:0/1835295930 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f481c0b5ab0 0x7f481c0b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.395+0000 7f482084f700 1 --2- 172.21.15.91:0/1835295930 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f481c06b620 0x7f481c1f8010 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.395+0000 7f482084f700 1 --2- 172.21.15.91:0/1835295930 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f481c0b4150 0x7f481c1f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.395+0000 7f482084f700 1 -- 172.21.15.91:0/1835295930 >> 172.21.15.91:0/1835295930 conn(0x7f481c0b2740 msgr2=0x7f481c0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:32.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.395+0000 7f482084f700 1 -- 172.21.15.91:0/1835295930 shutdown_connections 2023-11-24T06:09:32.400 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.395+0000 7f482084f700 1 -- 172.21.15.91:0/1835295930 wait complete. 2023-11-24T06:09:32.401 INFO:teuthology.orchestra.run.smithi091.stderr:instructed osd(s) 7 to deep-scrub 2023-11-24T06:09:32.411 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:09:32.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.527+0000 7f34c96b1700 1 Processor -- start 2023-11-24T06:09:32.531 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.527+0000 7f34c96b1700 1 -- start start 2023-11-24T06:09:32.534 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.528+0000 7f34c96b1700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f34c4143ae0 0x7f34c40b51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.534 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.528+0000 7f34c96b1700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f34c40722c0 0x7f34c406e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.534 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.528+0000 7f34c96b1700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c4143ec0 con 0x7f34c41440f0 2023-11-24T06:09:32.534 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.528+0000 7f34c96b1700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34c40b40a0 con 0x7f34c40722c0 2023-11-24T06:09:32.534 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.528+0000 7f34c96b1700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c40b3ae0 con 0x7f34c4143ae0 2023-11-24T06:09:32.534 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.528+0000 7f34c37fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f34c40722c0 0x7f34c406e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.534 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.528+0000 7f34c37fe700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f34c40722c0 0x7f34c406e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60624/0 (socket says 172.21.15.91:60624) 2023-11-24T06:09:32.535 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.528+0000 7f34c37fe700 1 -- 172.21.15.91:0/344763008 learned_addr learned my addr 172.21.15.91:0/344763008 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:32.535 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.528+0000 7f34c2ffd700 1 --2- 172.21.15.91:0/344763008 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f34c4143ae0 0x7f34c40b51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.535 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.529+0000 7f34c37fe700 1 -- 172.21.15.91:0/344763008 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f34c4143ae0 msgr2=0x7f34c40b51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:32.535 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.529+0000 7f34c37fe700 1 --2- 172.21.15.91:0/344763008 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f34c4143ae0 0x7f34c40b51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.535 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.529+0000 7f34c37fe700 1 -- 172.21.15.91:0/344763008 >> v1:172.21.15.94:6789/0 conn(0x7f34c41440f0 legacy=0x7f34c4071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.535 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.529+0000 7f34c37fe700 1 -- 172.21.15.91:0/344763008 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c40b3dc0 con 0x7f34c40722c0 2023-11-24T06:09:32.535 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.529+0000 7f34c1ffb700 1 -- 172.21.15.91:0/344763008 <== mon.1 v1:172.21.15.94:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891623925 0 0) 0x7f34c4143ec0 con 0x7f34c41440f0 2023-11-24T06:09:32.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.529+0000 7f34c37fe700 1 --2- 172.21.15.91:0/344763008 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f34c40722c0 0x7f34c406e510 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f34bc009e50 tx=0x7f34bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bdd8f71ce73dfb9 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:32.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.529+0000 7f34c1ffb700 1 -- 172.21.15.91:0/344763008 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc00e070 con 0x7f34c40722c0 2023-11-24T06:09:32.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.529+0000 7f34c1ffb700 1 -- 172.21.15.91:0/344763008 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34bc00b910 con 0x7f34c40722c0 2023-11-24T06:09:32.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.529+0000 7f34c1ffb700 1 -- 172.21.15.91:0/344763008 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc00bcc0 con 0x7f34c40722c0 2023-11-24T06:09:32.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.529+0000 7f34c96b1700 1 -- 172.21.15.91:0/344763008 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f34c40722c0 msgr2=0x7f34c406e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.529+0000 7f34c96b1700 1 --2- 172.21.15.91:0/344763008 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f34c40722c0 0x7f34c406e510 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f34bc009e50 tx=0x7f34bc00b330 comp rx=0 tx=0).stop 2023-11-24T06:09:32.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.529+0000 7f34c96b1700 1 -- 172.21.15.91:0/344763008 shutdown_connections 2023-11-24T06:09:32.536 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.529+0000 7f34c96b1700 1 --2- 172.21.15.91:0/344763008 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f34c40722c0 0x7f34c406e510 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.529+0000 7f34c96b1700 1 --2- 172.21.15.91:0/344763008 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f34c4143ae0 0x7f34c40b51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.529+0000 7f34c96b1700 1 -- 172.21.15.91:0/344763008 >> 172.21.15.91:0/344763008 conn(0x7f34c40b2740 msgr2=0x7f34c40b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:32.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.529+0000 7f34c96b1700 1 -- 172.21.15.91:0/344763008 shutdown_connections 2023-11-24T06:09:32.537 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.530+0000 7f34c96b1700 1 -- 172.21.15.91:0/344763008 wait complete. 2023-11-24T06:09:32.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.531+0000 7f34c96b1700 1 Processor -- start 2023-11-24T06:09:32.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.531+0000 7f34c96b1700 1 -- start start 2023-11-24T06:09:32.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.531+0000 7f34c96b1700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f34c40722c0 0x7f34c4136980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.531+0000 7f34c96b1700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f34c4143ae0 0x7f34c41f4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c96b1700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34c41440f0 0x7f34c41f4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.540 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c96b1700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c40cc560 con 0x7f34c40722c0 2023-11-24T06:09:32.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c96b1700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c4136ec0 con 0x7f34c41440f0 2023-11-24T06:09:32.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c96b1700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c40cb8e0 con 0x7f34c4143ae0 2023-11-24T06:09:32.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c2ffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f34c40722c0 0x7f34c4136980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c2ffd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f34c40722c0 0x7f34c4136980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60626/0 (socket says 172.21.15.91:60626) 2023-11-24T06:09:32.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c2ffd700 1 -- 172.21.15.91:0/2640456081 learned_addr learned my addr 172.21.15.91:0/2640456081 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:32.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c27fc700 1 --2- 172.21.15.91:0/2640456081 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f34c4143ae0 0x7f34c41f4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.541 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c37fe700 1 --2- 172.21.15.91:0/2640456081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34c41440f0 0x7f34c41f4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c2ffd700 1 -- 172.21.15.91:0/2640456081 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f34c4143ae0 msgr2=0x7f34c41f4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c2ffd700 1 --2- 172.21.15.91:0/2640456081 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f34c4143ae0 0x7f34c41f4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c2ffd700 1 -- 172.21.15.91:0/2640456081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34c41440f0 msgr2=0x7f34c41f4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c2ffd700 1 --2- 172.21.15.91:0/2640456081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34c41440f0 0x7f34c41f4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c2ffd700 1 -- 172.21.15.91:0/2640456081 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34bc009b30 con 0x7f34c40722c0 2023-11-24T06:09:32.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c37fe700 1 --2- 172.21.15.91:0/2640456081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34c41440f0 0x7f34c41f4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:32.542 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c2ffd700 1 --2- 172.21.15.91:0/2640456081 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f34c40722c0 0x7f34c4136980 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f34b4010bc0 tx=0x7f34b4010f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:32.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34ab7fe700 1 -- 172.21.15.91:0/2640456081 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b4009710 con 0x7f34c40722c0 2023-11-24T06:09:32.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34ab7fe700 1 -- 172.21.15.91:0/2640456081 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b400ee30 con 0x7f34c40722c0 2023-11-24T06:09:32.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c96b1700 1 -- 172.21.15.91:0/2640456081 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c40cbbc0 con 0x7f34c40722c0 2023-11-24T06:09:32.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34ab7fe700 1 -- 172.21.15.91:0/2640456081 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b4011650 con 0x7f34c40722c0 2023-11-24T06:09:32.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.532+0000 7f34c96b1700 1 -- 172.21.15.91:0/2640456081 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c40cc180 con 0x7f34c40722c0 2023-11-24T06:09:32.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.533+0000 7f34c96b1700 1 -- 172.21.15.91:0/2640456081 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c40cd8d0 con 0x7f34c40722c0 2023-11-24T06:09:32.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.534+0000 7f34ab7fe700 1 -- 172.21.15.91:0/2640456081 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f34b40117b0 con 0x7f34c40722c0 2023-11-24T06:09:32.543 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.534+0000 7f34ab7fe700 1 --2- 172.21.15.91:0/2640456081 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f34b006af60 0x7f34b006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:32.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.534+0000 7f34ab7fe700 1 -- 172.21.15.91:0/2640456081 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f34b408b690 con 0x7f34c40722c0 2023-11-24T06:09:32.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.534+0000 7f34c27fc700 1 --2- 172.21.15.91:0/2640456081 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f34b006af60 0x7f34b006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:32.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.534+0000 7f34c27fc700 1 --2- 172.21.15.91:0/2640456081 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f34b006af60 0x7f34b006d420 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f34c41f4590 tx=0x7f34b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:32.544 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.539+0000 7f34ab7fe700 1 -- 172.21.15.91:0/2640456081 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34b4056ae0 con 0x7f34c40722c0 2023-11-24T06:09:32.727 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.724+0000 7f34c96b1700 1 -- 172.21.15.91:0/2640456081 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f34c4144860 con 0x7f34b006af60 2023-11-24T06:09:32.729 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.726+0000 7f34ab7fe700 1 -- 172.21.15.91:0/2640456081 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50285 (secure 0 0 0) 0x7f34b80050d0 con 0x7f34b006af60 2023-11-24T06:09:32.730 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:32.732 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.728+0000 7f34c96b1700 1 -- 172.21.15.91:0/2640456081 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f34b006af60 msgr2=0x7f34b006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.732 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.728+0000 7f34c96b1700 1 --2- 172.21.15.91:0/2640456081 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f34b006af60 0x7f34b006d420 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f34c41f4590 tx=0x7f34b8009040 comp rx=0 tx=0).stop 2023-11-24T06:09:32.732 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.728+0000 7f34c96b1700 1 -- 172.21.15.91:0/2640456081 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f34c40722c0 msgr2=0x7f34c4136980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:32.732 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.728+0000 7f34c96b1700 1 --2- 172.21.15.91:0/2640456081 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f34c40722c0 0x7f34c4136980 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f34b4010bc0 tx=0x7f34b4010f80 comp rx=0 tx=0).stop 2023-11-24T06:09:32.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.728+0000 7f34c96b1700 1 -- 172.21.15.91:0/2640456081 shutdown_connections 2023-11-24T06:09:32.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.728+0000 7f34c96b1700 1 --2- 172.21.15.91:0/2640456081 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f34b006af60 0x7f34b006d420 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.728+0000 7f34c96b1700 1 --2- 172.21.15.91:0/2640456081 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f34c41440f0 0x7f34c41f4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.728+0000 7f34c96b1700 1 --2- 172.21.15.91:0/2640456081 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f34c40722c0 0x7f34c4136980 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.728+0000 7f34c96b1700 1 --2- 172.21.15.91:0/2640456081 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f34c4143ae0 0x7f34c41f4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:32.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.728+0000 7f34c96b1700 1 -- 172.21.15.91:0/2640456081 >> 172.21.15.91:0/2640456081 conn(0x7f34c40b2740 msgr2=0x7f34c415b380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:32.733 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.728+0000 7f34c96b1700 1 -- 172.21.15.91:0/2640456081 shutdown_connections 2023-11-24T06:09:32.734 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:32.728+0000 7f34c96b1700 1 -- 172.21.15.91:0/2640456081 wait complete. 2023-11-24T06:09:32.734 INFO:teuthology.orchestra.run.smithi091.stderr:dumped all 2023-11-24T06:09:32.744 INFO:teuthology.orchestra.run.smithi091.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2023-11-24T06:09:31.414875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":220724,"kb_used_data":4520,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753996,"statfs":{"total":773094113280,"available":772868091904,"internally_reserved":0,"allocated":4628480,"data_stored":2537184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-129,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.430586"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":180,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-11-24T06:09:22.986726+0000","last_change":"2023-11-24T06:07:49.463719+0000","last_active":"2023-11-24T06:09:22.986726+0000","last_peered":"2023-11-24T06:09:22.986726+0000","last_clean":"2023-11-24T06:09:22.986726+0000","last_became_active":"2023-11-24T06:07:49.462736+0000","last_became_peered":"2023-11-24T06:07:49.462736+0000","last_unstale":"2023-11-24T06:09:22.986726+0000","last_undegraded":"2023-11-24T06:09:22.986726+0000","last_fullsized":"2023-11-24T06:09:22.986726+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-24T06:07:47.860604+0000","last_clean_scrub_stamp":"2023-11-24T06:07:47.860604+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T11:31:16.508770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28028,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343812,"statfs":{"total":96636764160,"available":96608063488,"internally_reserved":0,"allocated":1024000,"data_stored":759924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46100000000000002}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51400000000000001}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.38},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.59899999999999998}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.56499999999999995}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.64900000000000002}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.61899999999999999}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.65900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":169556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.503}]},{"osd":1,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.53900000000000003}]},{"osd":2,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.55200000000000005}]},{"osd":3,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.435},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.46300000000000002}]},{"osd":4,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.38300000000000001}]},{"osd":5,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.44400000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.51200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574872,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27448,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":430080,"data_stored":169556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.7670000000000003,"5min":4.7670000000000003,"15min":4.7670000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.0810000000000004,"5min":6.0810000000000004,"15min":6.0810000000000004},"last":0.30099999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.8340000000000001,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.38900000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.34999999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.9529999999999998,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.70099999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.9640000000000004,"5min":5.9640000000000004,"15min":5.9640000000000004},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":3.02,"5min":3.02,"15min":3.02},"last":0.64700000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":6.048,"5min":6.048,"15min":6.048},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.9929999999999999,"5min":2.9929999999999999,"15min":2.9929999999999999},"last":0.69099999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":6.0129999999999999,"5min":6.0129999999999999,"15min":6.0129999999999999},"last":0.77800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":1024000,"data_stored":759924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":12.076000000000001,"5min":12.076000000000001,"15min":12.076000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":4.181,"5min":4.181,"15min":4.181},"last":0.41399999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.2089999999999996,"5min":4.2089999999999996,"15min":4.2089999999999996},"last":0.33900000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.38200000000000001}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":4.1520000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.442}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":4.1260000000000003,"5min":4.1260000000000003,"15min":4.1260000000000003},"last":0.48599999999999999}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.432}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.505}]}]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":169556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":9.7379999999999995,"5min":9.7379999999999995,"15min":9.7379999999999995},"last":0.38}]},{"osd":1,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":11.967000000000001,"5min":11.967000000000001,"15min":11.967000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":9.593,"5min":9.593,"15min":9.593},"last":0.33500000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.609,"5min":1.609,"15min":1.609},"last":0.27100000000000002}]},{"osd":4,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":9.6479999999999997,"5min":9.6479999999999997,"15min":9.6479999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":0.45100000000000001}]},{"osd":5,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.663,"5min":1.663,"15min":1.663},"last":0.55300000000000005},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":9.7119999999999997,"5min":9.7119999999999997,"15min":9.7119999999999997},"last":0.52800000000000002}]},{"osd":6,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.51200000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:08:46 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":9.6859999999999999,"5min":9.6859999999999999,"15min":9.6859999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.499}]}]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27448,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":430080,"data_stored":169556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.377},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.32600000000000001}]},{"osd":1,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.34499999999999997}]},{"osd":2,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.40000000000000002}]},{"osd":4,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.436}]},{"osd":5,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.501}]},{"osd":6,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.48899999999999999}]},{"osd":7,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.56100000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27448,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":430080,"data_stored":169556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.9539999999999997,"5min":6.9539999999999997,"15min":6.9539999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":7.0039999999999996,"5min":7.0039999999999996,"15min":7.0039999999999996},"last":0.71899999999999997}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":12.907,"5min":12.907,"15min":12.907},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":6.9710000000000001,"5min":6.9710000000000001,"15min":6.9710000000000001},"last":0.54800000000000004}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.78200000000000003}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":7.0540000000000003,"5min":7.0540000000000003,"15min":7.0540000000000003},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":7.0679999999999996,"5min":7.0679999999999996,"15min":7.0679999999999996},"last":0.67900000000000005}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.69899999999999995}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.749},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.86099999999999999}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.80000000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":169556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.495},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.65400000000000003}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.62},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.41899999999999998}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.996,"5min":1.996,"15min":1.996},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.52800000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.47299999999999998}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.72299999999999998}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.63900000000000001}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.70199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:09:32.747 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-11-24T06:07:47.860604+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=6, tm_min=7, tm_sec=47, tm_wday=4, tm_yday=328, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=24, tm_hour=6, tm_min=9, tm_sec=27, tm_wday=4, tm_yday=328, tm_isdst=0) 2023-11-24T06:09:32.748 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-11-24T06:09:52.748 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-24T06:09:52.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.880+0000 7ff424992700 1 Processor -- start 2023-11-24T06:09:52.883 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.880+0000 7ff424992700 1 -- start start 2023-11-24T06:09:52.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.881+0000 7ff424992700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff4200cfd00 0x7ff4200b4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:52.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.881+0000 7ff424992700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff4200c6070 0x7ff4200c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:52.889 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.881+0000 7ff424992700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4200c5be0 con 0x7ff4200d00e0 2023-11-24T06:09:52.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.881+0000 7ff424992700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4200cdad0 con 0x7ff4200cfd00 2023-11-24T06:09:52.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.881+0000 7ff424992700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4200cd230 con 0x7ff4200c6070 2023-11-24T06:09:52.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.881+0000 7ff41e59c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff4200cfd00 0x7ff4200b4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:52.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.881+0000 7ff41e59c700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7ff4200cfd00 0x7ff4200b4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60634/0 (socket says 172.21.15.91:60634) 2023-11-24T06:09:52.890 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.881+0000 7ff41e59c700 1 -- 172.21.15.91:0/2097685855 learned_addr learned my addr 172.21.15.91:0/2097685855 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:52.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.881+0000 7ff41ed9d700 1 --2- 172.21.15.91:0/2097685855 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff4200c6070 0x7ff4200c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:52.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.881+0000 7ff41e59c700 1 -- 172.21.15.91:0/2097685855 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff4200c6070 msgr2=0x7ff4200c6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:52.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.881+0000 7ff41e59c700 1 --2- 172.21.15.91:0/2097685855 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff4200c6070 0x7ff4200c6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:52.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.881+0000 7ff41e59c700 1 -- 172.21.15.91:0/2097685855 >> v1:172.21.15.94:6789/0 conn(0x7ff4200d00e0 legacy=0x7ff4200d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:52.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.881+0000 7ff41e59c700 1 -- 172.21.15.91:0/2097685855 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4200cd510 con 0x7ff4200cfd00 2023-11-24T06:09:52.891 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.881+0000 7ff41ed9d700 1 --2- 172.21.15.91:0/2097685855 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff4200c6070 0x7ff4200c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-11-24T06:09:52.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.881+0000 7ff41e59c700 1 --2- 172.21.15.91:0/2097685855 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff4200cfd00 0x7ff4200b4390 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff4100098f0 tx=0x7ff410009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2cc6ebfabeade8 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:52.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.882+0000 7ff41d59a700 1 -- 172.21.15.91:0/2097685855 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff410004230 con 0x7ff4200cfd00 2023-11-24T06:09:52.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.882+0000 7ff41d59a700 1 -- 172.21.15.91:0/2097685855 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff410004390 con 0x7ff4200cfd00 2023-11-24T06:09:52.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.882+0000 7ff41d59a700 1 -- 172.21.15.91:0/2097685855 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4100036a0 con 0x7ff4200cfd00 2023-11-24T06:09:52.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.882+0000 7ff424992700 1 -- 172.21.15.91:0/2097685855 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff4200cfd00 msgr2=0x7ff4200b4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:52.892 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.882+0000 7ff424992700 1 --2- 172.21.15.91:0/2097685855 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff4200cfd00 0x7ff4200b4390 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff4100098f0 tx=0x7ff410009c00 comp rx=0 tx=0).stop 2023-11-24T06:09:52.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.882+0000 7ff424992700 1 -- 172.21.15.91:0/2097685855 shutdown_connections 2023-11-24T06:09:52.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.882+0000 7ff424992700 1 --2- 172.21.15.91:0/2097685855 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff4200c6070 0x7ff4200c6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:52.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.882+0000 7ff424992700 1 --2- 172.21.15.91:0/2097685855 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff4200cfd00 0x7ff4200b4390 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:52.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.882+0000 7ff424992700 1 -- 172.21.15.91:0/2097685855 >> 172.21.15.91:0/2097685855 conn(0x7ff4200b3870 msgr2=0x7ff4200b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:52.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.882+0000 7ff424992700 1 -- 172.21.15.91:0/2097685855 shutdown_connections 2023-11-24T06:09:52.893 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.882+0000 7ff424992700 1 -- 172.21.15.91:0/2097685855 wait complete. 2023-11-24T06:09:52.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.884+0000 7ff424992700 1 Processor -- start 2023-11-24T06:09:52.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.884+0000 7ff424992700 1 -- start start 2023-11-24T06:09:52.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.884+0000 7ff424992700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff4200c6070 0x7ff4200b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:52.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.884+0000 7ff424992700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4200cfd00 0x7ff42014ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:52.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.884+0000 7ff424992700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff4200d00e0 0x7ff42014f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:52.894 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.884+0000 7ff424992700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42014f690 con 0x7ff4200d00e0 2023-11-24T06:09:52.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.884+0000 7ff424992700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4200ca1a0 con 0x7ff4200cfd00 2023-11-24T06:09:52.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.884+0000 7ff424992700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4200cb5b0 con 0x7ff4200c6070 2023-11-24T06:09:52.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.885+0000 7ff41ed9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff4200d00e0 0x7ff42014f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:52.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.885+0000 7ff41ed9d700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff4200d00e0 0x7ff42014f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60644/0 (socket says 172.21.15.91:60644) 2023-11-24T06:09:52.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.885+0000 7ff41ed9d700 1 -- 172.21.15.91:0/3908480613 learned_addr learned my addr 172.21.15.91:0/3908480613 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:52.895 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.885+0000 7ff41e59c700 1 --2- 172.21.15.91:0/3908480613 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff4200c6070 0x7ff4200b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:52.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.885+0000 7ff41dd9b700 1 --2- 172.21.15.91:0/3908480613 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4200cfd00 0x7ff42014ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:52.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.885+0000 7ff41ed9d700 1 -- 172.21.15.91:0/3908480613 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff4200c6070 msgr2=0x7ff4200b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:52.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.885+0000 7ff41ed9d700 1 --2- 172.21.15.91:0/3908480613 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff4200c6070 0x7ff4200b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:52.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.885+0000 7ff41ed9d700 1 -- 172.21.15.91:0/3908480613 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4200cfd00 msgr2=0x7ff42014ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:52.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.885+0000 7ff41ed9d700 1 --2- 172.21.15.91:0/3908480613 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4200cfd00 0x7ff42014ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:52.896 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.885+0000 7ff41ed9d700 1 -- 172.21.15.91:0/3908480613 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff410009550 con 0x7ff4200d00e0 2023-11-24T06:09:52.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.885+0000 7ff41ed9d700 1 --2- 172.21.15.91:0/3908480613 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff4200d00e0 0x7ff42014f150 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff41800cef0 tx=0x7ff41800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:52.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.885+0000 7ff40effd700 1 -- 172.21.15.91:0/3908480613 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41800ae50 con 0x7ff4200d00e0 2023-11-24T06:09:52.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.885+0000 7ff40effd700 1 -- 172.21.15.91:0/3908480613 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff418015440 con 0x7ff4200d00e0 2023-11-24T06:09:52.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.885+0000 7ff424992700 1 -- 172.21.15.91:0/3908480613 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4200cb8b0 con 0x7ff4200d00e0 2023-11-24T06:09:52.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.885+0000 7ff40effd700 1 -- 172.21.15.91:0/3908480613 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4180157f0 con 0x7ff4200d00e0 2023-11-24T06:09:52.897 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.885+0000 7ff424992700 1 -- 172.21.15.91:0/3908480613 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42015df50 con 0x7ff4200d00e0 2023-11-24T06:09:52.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.886+0000 7ff424992700 1 -- 172.21.15.91:0/3908480613 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff420067f60 con 0x7ff4200d00e0 2023-11-24T06:09:52.898 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.887+0000 7ff40effd700 1 -- 172.21.15.91:0/3908480613 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7ff41800d430 con 0x7ff4200d00e0 2023-11-24T06:09:52.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.887+0000 7ff40effd700 1 --2- 172.21.15.91:0/3908480613 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff40006af60 0x7ff40006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:52.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.887+0000 7ff40effd700 1 -- 172.21.15.91:0/3908480613 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff4180896e0 con 0x7ff4200d00e0 2023-11-24T06:09:52.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.887+0000 7ff41e59c700 1 --2- 172.21.15.91:0/3908480613 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff40006af60 0x7ff40006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:52.899 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.887+0000 7ff41e59c700 1 --2- 172.21.15.91:0/3908480613 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff40006af60 0x7ff40006d420 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff410009130 tx=0x7ff410008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:52.900 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:52.892+0000 7ff40effd700 1 -- 172.21.15.91:0/3908480613 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff418054b30 con 0x7ff4200d00e0 2023-11-24T06:09:53.080 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.077+0000 7ff424992700 1 -- 172.21.15.91:0/3908480613 --> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4200c5870 con 0x7ff40006af60 2023-11-24T06:09:53.082 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.079+0000 7ff40effd700 1 -- 172.21.15.91:0/3908480613 <== mgr.4107 v2:172.21.15.91:6832/3377926323 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50200 (secure 0 0 0) 0x7ff4100036a0 con 0x7ff40006af60 2023-11-24T06:09:53.083 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:53.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.081+0000 7ff424992700 1 -- 172.21.15.91:0/3908480613 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff40006af60 msgr2=0x7ff40006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:53.084 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.081+0000 7ff424992700 1 --2- 172.21.15.91:0/3908480613 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff40006af60 0x7ff40006d420 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff410009130 tx=0x7ff410008030 comp rx=0 tx=0).stop 2023-11-24T06:09:53.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.081+0000 7ff424992700 1 -- 172.21.15.91:0/3908480613 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff4200d00e0 msgr2=0x7ff42014f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:53.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.081+0000 7ff424992700 1 --2- 172.21.15.91:0/3908480613 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff4200d00e0 0x7ff42014f150 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff41800cef0 tx=0x7ff41800a340 comp rx=0 tx=0).stop 2023-11-24T06:09:53.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.081+0000 7ff424992700 1 -- 172.21.15.91:0/3908480613 shutdown_connections 2023-11-24T06:09:53.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.081+0000 7ff424992700 1 --2- 172.21.15.91:0/3908480613 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7ff40006af60 0x7ff40006d420 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.081+0000 7ff424992700 1 --2- 172.21.15.91:0/3908480613 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7ff4200d00e0 0x7ff42014f150 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.081+0000 7ff424992700 1 --2- 172.21.15.91:0/3908480613 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7ff4200c6070 0x7ff4200b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.081+0000 7ff424992700 1 --2- 172.21.15.91:0/3908480613 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7ff4200cfd00 0x7ff42014ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.085 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.081+0000 7ff424992700 1 -- 172.21.15.91:0/3908480613 >> 172.21.15.91:0/3908480613 conn(0x7ff4200b3870 msgr2=0x7ff4200c7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:53.086 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.081+0000 7ff424992700 1 -- 172.21.15.91:0/3908480613 shutdown_connections 2023-11-24T06:09:53.086 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.081+0000 7ff424992700 1 -- 172.21.15.91:0/3908480613 wait complete. 2023-11-24T06:09:53.086 INFO:teuthology.orchestra.run.smithi091.stderr:dumped all 2023-11-24T06:09:53.096 INFO:teuthology.orchestra.run.smithi091.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2023-11-24T06:09:51.418722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":220708,"kb_used_data":4520,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754012,"statfs":{"total":773094113280,"available":772868108288,"internally_reserved":0,"allocated":4628480,"data_stored":2537184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002221"},"pg_stats":[{"pgid":"1.0","version":"37'119","reported_seq":191,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-11-24T06:09:33.353316+0000","last_change":"2023-11-24T06:09:33.353355+0000","last_active":"2023-11-24T06:09:33.353316+0000","last_peered":"2023-11-24T06:09:33.353316+0000","last_clean":"2023-11-24T06:09:33.353316+0000","last_became_active":"2023-11-24T06:07:49.462736+0000","last_became_peered":"2023-11-24T06:07:49.462736+0000","last_unstale":"2023-11-24T06:09:33.353316+0000","last_undegraded":"2023-11-24T06:09:33.353316+0000","last_fullsized":"2023-11-24T06:09:33.353316+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'119","last_scrub_stamp":"2023-11-24T06:09:33.353211+0000","last_deep_scrub":"37'119","last_deep_scrub_stamp":"2023-11-24T06:09:33.353211+0000","last_clean_scrub_stamp":"2023-11-24T06:09:33.353211+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-25T15:36:01.686616+0000","scrub_duration":0.0060483380000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343824,"statfs":{"total":96636764160,"available":96608075776,"internally_reserved":0,"allocated":1024000,"data_stored":759924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.66300000000000003}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.73699999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.61399999999999999}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.753},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.57299999999999995}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.436},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.51200000000000001}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.495},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.47099999999999997}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.52900000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":169556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.66700000000000004}]},{"osd":1,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.52900000000000003}]},{"osd":2,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.59699999999999998}]},{"osd":3,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.56200000000000006}]},{"osd":4,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.441},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.57899999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.497}]},{"osd":7,"last update":"Fri Nov 24 06:08:52 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.624},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.65300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27448,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":430080,"data_stored":169556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":4.7670000000000003,"5min":4.7670000000000003,"15min":4.7670000000000003},"last":0.245},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":6.0810000000000004,"5min":6.0810000000000004,"15min":6.0810000000000004},"last":0.33100000000000002}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.2000000000000002,"5min":2.2000000000000002,"15min":2.2000000000000002},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.8340000000000001,"5min":3.8340000000000001,"15min":3.8340000000000001},"last":0.31}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.379}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.9529999999999998,"5min":2.9529999999999998,"15min":2.9529999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.64500000000000002}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":5.9640000000000004,"5min":5.9640000000000004,"15min":5.9640000000000004},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":3.02,"5min":3.02,"15min":3.02},"last":0.66000000000000003}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":6.048,"5min":6.048,"15min":6.048},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.9929999999999999,"5min":2.9929999999999999,"15min":2.9929999999999999},"last":0.68899999999999995}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":6.0129999999999999,"5min":6.0129999999999999,"15min":6.0129999999999999},"last":0.63200000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343824,"statfs":{"total":96636764160,"available":96608075776,"internally_reserved":0,"allocated":1024000,"data_stored":759924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":12.076000000000001,"5min":12.076000000000001,"15min":12.076000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":4.181,"5min":4.181,"15min":4.181},"last":0.48699999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":4.2089999999999996,"5min":4.2089999999999996,"15min":4.2089999999999996},"last":0.39400000000000002}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.443},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.52100000000000002}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":4.1520000000000001,"5min":4.1520000000000001,"15min":4.1520000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.58199999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":4.1260000000000003,"5min":4.1260000000000003,"15min":4.1260000000000003},"last":0.64100000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.505},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.65600000000000003}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.626}]}]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":169556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.36099999999999999,"15min":0.36499999999999999},"min":{"1min":0.23599999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60799999999999998,"5min":1.579,"15min":1.579},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.70599999999999996,"15min":0.76000000000000001},"min":{"1min":0.26100000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.69299999999999995,"5min":9.7379999999999995,"15min":9.7379999999999995},"last":0.48799999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.81299999999999994,"15min":0.88700000000000001},"min":{"1min":0.247,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.54000000000000004,"5min":11.967000000000001,"15min":11.967000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.72399999999999998,"15min":0.77800000000000002},"min":{"1min":0.28199999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.80700000000000005,"5min":9.593,"15min":9.593},"last":0.56000000000000005}]},{"osd":3,"last update":"Fri Nov 24 06:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.435,"15min":0.437},"min":{"1min":0.29899999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.70999999999999996,"5min":1.417,"15min":1.417},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.43099999999999999,"15min":0.436},"min":{"1min":0.27100000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.88600000000000001,"5min":1.609,"15min":1.609},"last":0.53900000000000003}]},{"osd":4,"last update":"Fri Nov 24 06:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.96199999999999997,"15min":0.99199999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":5.0380000000000003,"5min":9.6479999999999997,"15min":9.6479999999999997},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.627,"5min":0.63700000000000001,"15min":0.63900000000000001},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.2569999999999999,"5min":1.448,"15min":1.448},"last":0.57799999999999996}]},{"osd":5,"last update":"Fri Nov 24 06:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.624,"15min":0.628},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.056,"5min":1.663,"15min":1.663},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.878,"5min":1.01,"15min":1.032},"min":{"1min":0.46999999999999997,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":6.2590000000000003,"5min":9.7119999999999997,"15min":9.7119999999999997},"last":0.63800000000000001}]},{"osd":6,"last update":"Fri Nov 24 06:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.64600000000000002,"15min":0.65100000000000002},"min":{"1min":0.45500000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.95099999999999996,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.44900000000000001,"5min":0.434,"15min":0.434},"max":{"1min":1.1819999999999999,"5min":1.637,"15min":1.637},"last":0.65400000000000003}]},{"osd":7,"last update":"Fri Nov 24 06:09:47 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.96399999999999997,"15min":1.016},"min":{"1min":0.48399999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1459999999999999,"5min":9.6859999999999999,"15min":9.6859999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.66400000000000003,"15min":0.66700000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.107,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.60499999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27448,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":430080,"data_stored":169556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.35499999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.42699999999999999}]},{"osd":2,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.377}]},{"osd":4,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.65300000000000002}]},{"osd":5,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.55100000000000005}]},{"osd":6,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.628}]},{"osd":7,"last update":"Fri Nov 24 06:08:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.57399999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27448,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":430080,"data_stored":169556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.9539999999999997,"5min":6.9539999999999997,"15min":6.9539999999999997},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":7.0039999999999996,"5min":7.0039999999999996,"15min":7.0039999999999996},"last":0.60599999999999998}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":12.907,"5min":12.907,"15min":12.907},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":6.9710000000000001,"5min":6.9710000000000001,"15min":6.9710000000000001},"last":0.624}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.497},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.65900000000000003}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":7.0540000000000003,"5min":7.0540000000000003,"15min":7.0540000000000003},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":7.0679999999999996,"5min":7.0679999999999996,"15min":7.0679999999999996},"last":0.63500000000000001}]},{"osd":5,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.58199999999999996}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.745}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.67700000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":169556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.73699999999999999}]},{"osd":1,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.4079999999999999,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.996,"5min":1.996,"15min":1.996},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.76200000000000001}]},{"osd":3,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60899999999999999}]},{"osd":4,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.41699999999999998}]},{"osd":6,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.51900000000000002}]},{"osd":7,"last update":"Fri Nov 24 06:08:47 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.503},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.47699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-24T06:09:53.098 DEBUG:teuthology.orchestra.run.smithi091:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-11-24T06:09:53.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.215+0000 7f274e1cd700 1 Processor -- start 2023-11-24T06:09:53.218 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.215+0000 7f274e1cd700 1 -- start start 2023-11-24T06:09:53.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f274e1cd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f27480b4150 0x7f274806de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:53.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f274e1cd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f27480b5ab0 0x7f274806e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:53.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f274e1cd700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2748071f50 con 0x7f274806b620 2023-11-24T06:09:53.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f274e1cd700 1 -- --> v2:172.21.15.91:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2748072230 con 0x7f27480b4150 2023-11-24T06:09:53.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f274e1cd700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2748072510 con 0x7f27480b5ab0 2023-11-24T06:09:53.221 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f2746ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f27480b4150 0x7f274806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:53.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f27477fe700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7f274806b620 0x7f27480b3a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55582/0 (socket says 172.21.15.91:55582) 2023-11-24T06:09:53.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f2746ffd700 1 --2- >> v2:172.21.15.91:3300/0 conn(0x7f27480b4150 0x7f274806de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60650/0 (socket says 172.21.15.91:60650) 2023-11-24T06:09:53.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f27477fe700 1 -- 172.21.15.91:0/107381109 learned_addr learned my addr 172.21.15.91:0/107381109 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:53.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f2747fff700 1 --2- 172.21.15.91:0/107381109 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f27480b5ab0 0x7f274806e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:53.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f2746ffd700 1 -- 172.21.15.91:0/107381109 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f27480b5ab0 msgr2=0x7f274806e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:53.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f2746ffd700 1 --2- 172.21.15.91:0/107381109 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f27480b5ab0 0x7f274806e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.222 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f2746ffd700 1 -- 172.21.15.91:0/107381109 >> v1:172.21.15.91:6789/0 conn(0x7f274806b620 legacy=0x7f27480b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:53.223 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f2746ffd700 1 -- 172.21.15.91:0/107381109 --> v2:172.21.15.91:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27480727f0 con 0x7f27480b4150 2023-11-24T06:09:53.223 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f27467fc700 1 -- 172.21.15.91:0/107381109 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373113739 0 0) 0x7f2748071f50 con 0x7f274806b620 2023-11-24T06:09:53.223 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f2746ffd700 1 --2- 172.21.15.91:0/107381109 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27480b4150 0x7f274806de50 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2738009840 tx=0x7f2738009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d250df90adcaf97d server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:53.223 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.216+0000 7f27467fc700 1 -- 172.21.15.91:0/107381109 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738011830 con 0x7f27480b4150 2023-11-24T06:09:53.223 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.217+0000 7f27467fc700 1 -- 172.21.15.91:0/107381109 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2738011990 con 0x7f27480b4150 2023-11-24T06:09:53.223 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.217+0000 7f27467fc700 1 -- 172.21.15.91:0/107381109 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738011d20 con 0x7f27480b4150 2023-11-24T06:09:53.224 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.217+0000 7f274e1cd700 1 -- 172.21.15.91:0/107381109 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27480b4150 msgr2=0x7f274806de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:53.224 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.217+0000 7f274e1cd700 1 --2- 172.21.15.91:0/107381109 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27480b4150 0x7f274806de50 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f2738009840 tx=0x7f2738009b50 comp rx=0 tx=0).stop 2023-11-24T06:09:53.224 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.217+0000 7f274e1cd700 1 -- 172.21.15.91:0/107381109 shutdown_connections 2023-11-24T06:09:53.224 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.217+0000 7f274e1cd700 1 --2- 172.21.15.91:0/107381109 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f27480b5ab0 0x7f274806e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.224 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.217+0000 7f274e1cd700 1 --2- 172.21.15.91:0/107381109 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27480b4150 0x7f274806de50 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.224 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.217+0000 7f274e1cd700 1 -- 172.21.15.91:0/107381109 >> 172.21.15.91:0/107381109 conn(0x7f27480b2740 msgr2=0x7f27480b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:53.224 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.217+0000 7f274e1cd700 1 -- 172.21.15.91:0/107381109 shutdown_connections 2023-11-24T06:09:53.224 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.217+0000 7f274e1cd700 1 -- 172.21.15.91:0/107381109 wait complete. 2023-11-24T06:09:53.225 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.218+0000 7f274e1cd700 1 Processor -- start 2023-11-24T06:09:53.227 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.218+0000 7f274e1cd700 1 -- start start 2023-11-24T06:09:53.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.219+0000 7f274e1cd700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f274806b620 0x7f27481e4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:53.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.219+0000 7f274e1cd700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27480b4150 0x7f27480c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:53.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.219+0000 7f274e1cd700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27480b5ab0 0x7f27480ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:53.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.219+0000 7f274e1cd700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27480c8dc0 con 0x7f27480b5ab0 2023-11-24T06:09:53.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.219+0000 7f274e1cd700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27480c73e0 con 0x7f27480b4150 2023-11-24T06:09:53.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.219+0000 7f274e1cd700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27480c76c0 con 0x7f274806b620 2023-11-24T06:09:53.228 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.219+0000 7f2747fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27480b5ab0 0x7f27480ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:53.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.219+0000 7f2747fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27480b5ab0 0x7f27480ae680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60658/0 (socket says 172.21.15.91:60658) 2023-11-24T06:09:53.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.219+0000 7f2747fff700 1 -- 172.21.15.91:0/1948952571 learned_addr learned my addr 172.21.15.91:0/1948952571 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:53.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.220+0000 7f27477fe700 1 --2- 172.21.15.91:0/1948952571 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f274806b620 0x7f27481e4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:53.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.220+0000 7f2746ffd700 1 --2- 172.21.15.91:0/1948952571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27480b4150 0x7f27480c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:53.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.220+0000 7f2747fff700 1 -- 172.21.15.91:0/1948952571 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f274806b620 msgr2=0x7f27481e4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:53.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.220+0000 7f2747fff700 1 --2- 172.21.15.91:0/1948952571 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f274806b620 0x7f27481e4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.229 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.220+0000 7f2747fff700 1 -- 172.21.15.91:0/1948952571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27480b4150 msgr2=0x7f27480c6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:53.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.220+0000 7f2747fff700 1 --2- 172.21.15.91:0/1948952571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27480b4150 0x7f27480c6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.220+0000 7f2747fff700 1 -- 172.21.15.91:0/1948952571 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27380094a0 con 0x7f27480b5ab0 2023-11-24T06:09:53.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.220+0000 7f2747fff700 1 --2- 172.21.15.91:0/1948952571 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27480b5ab0 0x7f27480ae680 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2740007e60 tx=0x7f274000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:53.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.220+0000 7f272ffff700 1 -- 172.21.15.91:0/1948952571 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274000ede0 con 0x7f27480b5ab0 2023-11-24T06:09:53.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.220+0000 7f272ffff700 1 -- 172.21.15.91:0/1948952571 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2740014420 con 0x7f27480b5ab0 2023-11-24T06:09:53.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.220+0000 7f274e1cd700 1 -- 172.21.15.91:0/1948952571 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27480c79a0 con 0x7f27480b5ab0 2023-11-24T06:09:53.230 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.220+0000 7f272ffff700 1 -- 172.21.15.91:0/1948952571 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27400147d0 con 0x7f27480b5ab0 2023-11-24T06:09:53.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.220+0000 7f274e1cd700 1 -- 172.21.15.91:0/1948952571 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27480c6f40 con 0x7f27480b5ab0 2023-11-24T06:09:53.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.221+0000 7f274e1cd700 1 -- 172.21.15.91:0/1948952571 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2748075480 con 0x7f27480b5ab0 2023-11-24T06:09:53.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.221+0000 7f272ffff700 1 -- 172.21.15.91:0/1948952571 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7f2740014930 con 0x7f27480b5ab0 2023-11-24T06:09:53.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.222+0000 7f272ffff700 1 --2- 172.21.15.91:0/1948952571 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f273406af60 0x7f273406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:53.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.222+0000 7f272ffff700 1 -- 172.21.15.91:0/1948952571 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f27400899d0 con 0x7f27480b5ab0 2023-11-24T06:09:53.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.222+0000 7f27477fe700 1 --2- 172.21.15.91:0/1948952571 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f273406af60 0x7f273406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:53.231 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.222+0000 7f27477fe700 1 --2- 172.21.15.91:0/1948952571 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f273406af60 0x7f273406d420 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f273c000f60 tx=0x7f273c007be0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:53.232 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.226+0000 7f272ffff700 1 -- 172.21.15.91:0/1948952571 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2740054e20 con 0x7f27480b5ab0 2023-11-24T06:09:53.418 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.415+0000 7f274e1cd700 1 -- 172.21.15.91:0/1948952571 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f27480c8ba0 con 0x7f27480b5ab0 2023-11-24T06:09:53.421 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.419+0000 7f272ffff700 1 -- 172.21.15.91:0/1948952571 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v12)=0 v12) v1 ==== 123+0+0 (secure 0 0 0) 0x7f2740058440 con 0x7f27480b5ab0 2023-11-24T06:09:53.423 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.420+0000 7f272ffff700 1 -- 172.21.15.91:0/1948952571 <== mon.0 v2:172.21.15.91:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f2740054e20 con 0x7f27480b5ab0 2023-11-24T06:09:53.424 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.421+0000 7f274e1cd700 1 -- 172.21.15.91:0/1948952571 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f273406af60 msgr2=0x7f273406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:53.424 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.421+0000 7f274e1cd700 1 --2- 172.21.15.91:0/1948952571 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f273406af60 0x7f273406d420 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f273c000f60 tx=0x7f273c007be0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.424 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.421+0000 7f274e1cd700 1 -- 172.21.15.91:0/1948952571 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27480b5ab0 msgr2=0x7f27480ae680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:53.425 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.421+0000 7f274e1cd700 1 --2- 172.21.15.91:0/1948952571 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27480b5ab0 0x7f27480ae680 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2740007e60 tx=0x7f274000e3c0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.425 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.421+0000 7f274e1cd700 1 -- 172.21.15.91:0/1948952571 shutdown_connections 2023-11-24T06:09:53.425 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.421+0000 7f274e1cd700 1 --2- 172.21.15.91:0/1948952571 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7f273406af60 0x7f273406d420 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.425 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.421+0000 7f274e1cd700 1 --2- 172.21.15.91:0/1948952571 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7f27480b5ab0 0x7f27480ae680 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.425 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.421+0000 7f274e1cd700 1 --2- 172.21.15.91:0/1948952571 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7f274806b620 0x7f27481e4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.425 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.421+0000 7f274e1cd700 1 --2- 172.21.15.91:0/1948952571 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7f27480b4150 0x7f27480c6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.426 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.421+0000 7f274e1cd700 1 -- 172.21.15.91:0/1948952571 >> 172.21.15.91:0/1948952571 conn(0x7f27480b2740 msgr2=0x7f27481e79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:53.426 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.421+0000 7f274e1cd700 1 -- 172.21.15.91:0/1948952571 shutdown_connections 2023-11-24T06:09:53.426 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.422+0000 7f274e1cd700 1 -- 172.21.15.91:0/1948952571 wait complete. 2023-11-24T06:09:53.437 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-11-24T06:09:53.437 DEBUG:teuthology.orchestra.run.smithi091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-11-24T06:09:53.556 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.554+0000 7fad7e8c1700 1 Processor -- start 2023-11-24T06:09:53.557 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.554+0000 7fad7e8c1700 1 -- start start 2023-11-24T06:09:53.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad7e8c1700 1 --2- >> v2:172.21.15.94:3300/0 conn(0x7fad780b5ab0 0x7fad7806e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:53.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad7e8c1700 1 -- --> v1:172.21.15.91:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad78071f50 con 0x7fad7806b620 2023-11-24T06:09:53.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad7e8c1700 1 -- --> v1:172.21.15.94:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad78072230 con 0x7fad780b4150 2023-11-24T06:09:53.563 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad7e8c1700 1 -- --> v2:172.21.15.94:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad78072510 con 0x7fad780b5ab0 2023-11-24T06:09:53.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad77fff700 1 --1- >> v1:172.21.15.91:6789/0 conn(0x7fad7806b620 0x7fad780b3a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.91:6789/0 says I am v1:172.21.15.91:55596/0 (socket says 172.21.15.91:55596) 2023-11-24T06:09:53.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad77fff700 1 -- 172.21.15.91:0/2458342758 learned_addr learned my addr 172.21.15.91:0/2458342758 (peer_addr_for_me v1:172.21.15.91:0/0) 2023-11-24T06:09:53.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad7ce5e700 1 --2- 172.21.15.91:0/2458342758 >> v2:172.21.15.94:3300/0 conn(0x7fad780b5ab0 0x7fad7806e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:53.564 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad76ffd700 1 -- 172.21.15.91:0/2458342758 <== mon.0 v1:172.21.15.91:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367847863 0 0) 0x7fad78071f50 con 0x7fad7806b620 2023-11-24T06:09:53.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad76ffd700 1 -- 172.21.15.91:0/2458342758 --> v1:172.21.15.91:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad600035b0 con 0x7fad7806b620 2023-11-24T06:09:53.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad76ffd700 1 -- 172.21.15.91:0/2458342758 <== mon.0 v1:172.21.15.91:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2822848540 0 0) 0x7fad600035b0 con 0x7fad7806b620 2023-11-24T06:09:53.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad76ffd700 1 -- 172.21.15.91:0/2458342758 >> v2:172.21.15.94:3300/0 conn(0x7fad780b5ab0 msgr2=0x7fad7806e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:53.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad76ffd700 1 --2- 172.21.15.91:0/2458342758 >> v2:172.21.15.94:3300/0 conn(0x7fad780b5ab0 0x7fad7806e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.565 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad76ffd700 1 -- 172.21.15.91:0/2458342758 >> v1:172.21.15.94:6789/0 conn(0x7fad780b4150 legacy=0x7fad7806de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:53.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad76ffd700 1 -- 172.21.15.91:0/2458342758 --> v1:172.21.15.91:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad780727f0 con 0x7fad7806b620 2023-11-24T06:09:53.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad76ffd700 1 -- 172.21.15.91:0/2458342758 <== mon.0 v1:172.21.15.91:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3369851059 0 0) 0x7fad68001670 con 0x7fad7806b620 2023-11-24T06:09:53.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad76ffd700 1 -- 172.21.15.91:0/2458342758 >> v1:172.21.15.91:6789/0 conn(0x7fad7806b620 legacy=0x7fad780b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:53.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad76ffd700 1 --2- 172.21.15.91:0/2458342758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad60003a00 0x7fad60003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:53.566 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad76ffd700 1 --2- 172.21.15.91:0/2458342758 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad60004850 0x7fad60004cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:53.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad76ffd700 1 --2- 172.21.15.91:0/2458342758 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad60005210 0x7fad6000ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:53.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad76ffd700 1 -- 172.21.15.91:0/2458342758 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad78072510 con 0x7fad60005210 2023-11-24T06:09:53.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad76ffd700 1 -- 172.21.15.91:0/2458342758 --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6000c0b0 con 0x7fad60003a00 2023-11-24T06:09:53.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad76ffd700 1 -- 172.21.15.91:0/2458342758 --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6000c270 con 0x7fad60004850 2023-11-24T06:09:53.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad7ce5e700 1 --2- 172.21.15.91:0/2458342758 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad60005210 0x7fad6000ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:53.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.555+0000 7fad77fff700 1 --2- 172.21.15.91:0/2458342758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad60003a00 0x7fad60003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:53.567 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.556+0000 7fad777fe700 1 --2- 172.21.15.91:0/2458342758 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad60004850 0x7fad60004cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:53.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.556+0000 7fad7ce5e700 1 -- 172.21.15.91:0/2458342758 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad60004850 msgr2=0x7fad60004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:53.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.556+0000 7fad7ce5e700 1 --2- 172.21.15.91:0/2458342758 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad60004850 0x7fad60004cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.556+0000 7fad7ce5e700 1 -- 172.21.15.91:0/2458342758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad60003a00 msgr2=0x7fad60003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-11-24T06:09:53.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.556+0000 7fad7ce5e700 1 --2- 172.21.15.91:0/2458342758 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad60003a00 0x7fad60003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.556+0000 7fad7ce5e700 1 -- 172.21.15.91:0/2458342758 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad600035b0 con 0x7fad60005210 2023-11-24T06:09:53.568 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.556+0000 7fad7ce5e700 1 -- 172.21.15.91:0/2458342758 reap_dead start 2023-11-24T06:09:53.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.556+0000 7fad7ce5e700 1 --2- 172.21.15.91:0/2458342758 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad60005210 0x7fad6000ba10 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fad70009fc0 tx=0x7fad60005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40ea6b26b8d0184 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:53.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.556+0000 7fad76ffd700 1 -- 172.21.15.91:0/2458342758 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad70003a20 con 0x7fad60005210 2023-11-24T06:09:53.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.556+0000 7fad76ffd700 1 -- 172.21.15.91:0/2458342758 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fad7000f040 con 0x7fad60005210 2023-11-24T06:09:53.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.556+0000 7fad76ffd700 1 -- 172.21.15.91:0/2458342758 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad70003dc0 con 0x7fad60005210 2023-11-24T06:09:53.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.556+0000 7fad7e8c1700 1 -- 172.21.15.91:0/2458342758 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad60005210 msgr2=0x7fad6000ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:53.569 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.556+0000 7fad7e8c1700 1 --2- 172.21.15.91:0/2458342758 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad60005210 0x7fad6000ba10 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fad70009fc0 tx=0x7fad60005880 comp rx=0 tx=0).stop 2023-11-24T06:09:53.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.557+0000 7fad7e8c1700 1 -- 172.21.15.91:0/2458342758 shutdown_connections 2023-11-24T06:09:53.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.557+0000 7fad7e8c1700 1 --2- 172.21.15.91:0/2458342758 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad60005210 0x7fad6000ba10 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.557+0000 7fad7e8c1700 1 -- 172.21.15.91:0/2458342758 >> 172.21.15.91:0/2458342758 conn(0x7fad780b2740 msgr2=0x7fad780b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:53.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.557+0000 7fad7e8c1700 1 -- 172.21.15.91:0/2458342758 shutdown_connections 2023-11-24T06:09:53.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.557+0000 7fad7e8c1700 1 -- 172.21.15.91:0/2458342758 wait complete. 2023-11-24T06:09:53.570 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.558+0000 7fad7e8c1700 1 Processor -- start 2023-11-24T06:09:53.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.558+0000 7fad7e8c1700 1 -- start start 2023-11-24T06:09:53.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad7e8c1700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad60005210 0x7fad780c9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:53.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad7e8c1700 1 --2- >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad781f7fa0 0x7fad780c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:53.571 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad7e8c1700 1 --2- >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad7806e840 0x7fad7806ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:53.572 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad7e8c1700 1 -- --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7806f230 con 0x7fad60005210 2023-11-24T06:09:53.572 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad7e8c1700 1 -- --> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad78134e80 con 0x7fad7806e840 2023-11-24T06:09:53.572 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad7e8c1700 1 -- --> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad780c68e0 con 0x7fad781f7fa0 2023-11-24T06:09:53.572 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad77fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad60005210 0x7fad780c9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:53.572 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad77fff700 1 --2- >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad60005210 0x7fad780c9520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.91:3300/0 says I am v2:172.21.15.91:60674/0 (socket says 172.21.15.91:60674) 2023-11-24T06:09:53.572 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad77fff700 1 -- 172.21.15.91:0/886391175 learned_addr learned my addr 172.21.15.91:0/886391175 (peer_addr_for_me v2:172.21.15.91:0/0) 2023-11-24T06:09:53.573 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad7ce5e700 1 --2- 172.21.15.91:0/886391175 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad7806e840 0x7fad7806ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:53.573 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad777fe700 1 --2- 172.21.15.91:0/886391175 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad781f7fa0 0x7fad780c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:53.573 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad77fff700 1 -- 172.21.15.91:0/886391175 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad781f7fa0 msgr2=0x7fad780c63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:53.573 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad77fff700 1 --2- 172.21.15.91:0/886391175 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad781f7fa0 0x7fad780c63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.573 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad77fff700 1 -- 172.21.15.91:0/886391175 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad7806e840 msgr2=0x7fad7806ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:53.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad77fff700 1 --2- 172.21.15.91:0/886391175 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad7806e840 0x7fad7806ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad77fff700 1 -- 172.21.15.91:0/886391175 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad781350d0 con 0x7fad60005210 2023-11-24T06:09:53.574 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad777fe700 1 --2- 172.21.15.91:0/886391175 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad781f7fa0 0x7fad780c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-24T06:09:53.577 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad77fff700 1 --2- 172.21.15.91:0/886391175 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad60005210 0x7fad780c9520 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fad680072f0 tx=0x7fad68007600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:53.577 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad74ff9700 1 -- 172.21.15.91:0/886391175 <== mon.0 v2:172.21.15.91:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad68011580 con 0x7fad60005210 2023-11-24T06:09:53.577 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad74ff9700 1 -- 172.21.15.91:0/886391175 <== mon.0 v2:172.21.15.91:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fad68011700 con 0x7fad60005210 2023-11-24T06:09:53.578 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad7e8c1700 1 -- 172.21.15.91:0/886391175 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad781353b0 con 0x7fad60005210 2023-11-24T06:09:53.578 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad74ff9700 1 -- 172.21.15.91:0/886391175 <== mon.0 v2:172.21.15.91:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad68011ad0 con 0x7fad60005210 2023-11-24T06:09:53.578 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.559+0000 7fad7e8c1700 1 -- 172.21.15.91:0/886391175 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad78135970 con 0x7fad60005210 2023-11-24T06:09:53.578 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.560+0000 7fad7e8c1700 1 -- 172.21.15.91:0/886391175 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad78068670 con 0x7fad60005210 2023-11-24T06:09:53.579 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.561+0000 7fad74ff9700 1 -- 172.21.15.91:0/886391175 <== mon.0 v2:172.21.15.91:3300/0 4 ==== mgrmap(e 6) v1 ==== 89282+0+0 (secure 0 0 0) 0x7fad68011c30 con 0x7fad60005210 2023-11-24T06:09:53.579 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.561+0000 7fad74ff9700 1 --2- 172.21.15.91:0/886391175 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad6406af60 0x7fad6406d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-24T06:09:53.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.561+0000 7fad74ff9700 1 -- 172.21.15.91:0/886391175 <== mon.0 v2:172.21.15.91:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fad68089e10 con 0x7fad60005210 2023-11-24T06:09:53.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.562+0000 7fad777fe700 1 --2- 172.21.15.91:0/886391175 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad6406af60 0x7fad6406d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-24T06:09:53.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.562+0000 7fad777fe700 1 --2- 172.21.15.91:0/886391175 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad6406af60 0x7fad6406d420 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fad78144a60 tx=0x7fad6c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-24T06:09:53.580 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.565+0000 7fad74ff9700 1 -- 172.21.15.91:0/886391175 <== mon.0 v2:172.21.15.91:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad680551c0 con 0x7fad60005210 2023-11-24T06:09:53.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.796+0000 7fad7e8c1700 1 -- 172.21.15.91:0/886391175 --> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad780c6b10 con 0x7fad60005210 2023-11-24T06:09:53.799 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.797+0000 7fad74ff9700 1 -- 172.21.15.91:0/886391175 <== mon.0 v2:172.21.15.91:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3693 (secure 0 0 0) 0x7fad680587e0 con 0x7fad60005210 2023-11-24T06:09:53.800 INFO:teuthology.orchestra.run.smithi091.stdout: 2023-11-24T06:09:53.800 INFO:teuthology.orchestra.run.smithi091.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4706,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6834/1685174039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6832","nonce":1685174039},{"type":"v1","addr":"172.21.15.94:6834","nonce":1685174039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4709,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.94:6835/1508344631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.94:6833","nonce":1508344631},{"type":"v1","addr":"172.21.15.94:6835","nonce":1508344631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5064,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6836/28114058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6834","nonce":28114058},{"type":"v1","addr":"172.21.15.91:6836","nonce":28114058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.91:6837/2730466428","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.91:6835","nonce":2730466428},{"type":"v1","addr":"172.21.15.91:6837","nonce":2730466428}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2023-11-24T06:09:53.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.799+0000 7fad7e8c1700 1 -- 172.21.15.91:0/886391175 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad6406af60 msgr2=0x7fad6406d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:53.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.799+0000 7fad7e8c1700 1 --2- 172.21.15.91:0/886391175 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad6406af60 0x7fad6406d420 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fad78144a60 tx=0x7fad6c009040 comp rx=0 tx=0).stop 2023-11-24T06:09:53.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.799+0000 7fad7e8c1700 1 -- 172.21.15.91:0/886391175 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad60005210 msgr2=0x7fad780c9520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-24T06:09:53.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.799+0000 7fad7e8c1700 1 --2- 172.21.15.91:0/886391175 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad60005210 0x7fad780c9520 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fad680072f0 tx=0x7fad68007600 comp rx=0 tx=0).stop 2023-11-24T06:09:53.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.799+0000 7fad7e8c1700 1 -- 172.21.15.91:0/886391175 shutdown_connections 2023-11-24T06:09:53.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.799+0000 7fad7e8c1700 1 --2- 172.21.15.91:0/886391175 >> [v2:172.21.15.91:6832/3377926323,v1:172.21.15.91:6833/3377926323] conn(0x7fad6406af60 0x7fad6406d420 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.802 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.799+0000 7fad7e8c1700 1 --2- 172.21.15.91:0/886391175 >> [v2:172.21.15.94:3300/0,v1:172.21.15.94:6789/0] conn(0x7fad7806e840 0x7fad7806ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.799+0000 7fad7e8c1700 1 --2- 172.21.15.91:0/886391175 >> [v2:172.21.15.91:3300/0,v1:172.21.15.91:6789/0] conn(0x7fad60005210 0x7fad780c9520 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.799+0000 7fad7e8c1700 1 --2- 172.21.15.91:0/886391175 >> [v2:172.21.15.94:3301/0,v1:172.21.15.94:6790/0] conn(0x7fad781f7fa0 0x7fad780c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-24T06:09:53.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.799+0000 7fad7e8c1700 1 -- 172.21.15.91:0/886391175 >> 172.21.15.91:0/886391175 conn(0x7fad780b2740 msgr2=0x7fad7806de30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-24T06:09:53.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.799+0000 7fad7e8c1700 1 -- 172.21.15.91:0/886391175 shutdown_connections 2023-11-24T06:09:53.803 INFO:teuthology.orchestra.run.smithi091.stderr:2023-11-24T06:09:53.799+0000 7fad7e8c1700 1 -- 172.21.15.91:0/886391175 wait complete. 2023-11-24T06:09:53.803 INFO:teuthology.orchestra.run.smithi091.stderr:dumped fsmap epoch 19 2023-11-24T06:09:53.814 DEBUG:tasks.cephfs.filesystem:already dead... 2023-11-24T06:09:53.814 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-24T06:09:53.815 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-11-24T06:09:53.815 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:53.823 INFO:tasks.ceph.mds.a:Stopped 2023-11-24T06:09:53.823 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-11-24T06:09:53.823 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:53.872 INFO:tasks.ceph.mds.c:Stopped 2023-11-24T06:09:53.873 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-11-24T06:09:53.873 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:53.883 INFO:tasks.ceph.mds.b:Stopped 2023-11-24T06:09:53.884 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-11-24T06:09:53.884 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:53.935 INFO:tasks.ceph.mds.d:Stopped 2023-11-24T06:09:53.935 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-24T06:09:53.935 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-24T06:09:53.935 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:53.973 INFO:tasks.ceph.osd.0:Stopped 2023-11-24T06:09:53.974 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-24T06:09:53.974 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:54.040 INFO:tasks.ceph.osd.1:Stopped 2023-11-24T06:09:54.040 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-24T06:09:54.040 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:54.114 INFO:tasks.ceph.osd.2:Stopped 2023-11-24T06:09:54.115 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-11-24T06:09:54.115 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:54.188 INFO:tasks.ceph.osd.3:Stopped 2023-11-24T06:09:54.189 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-11-24T06:09:54.189 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:54.233 INFO:tasks.ceph.osd.4:Stopped 2023-11-24T06:09:54.233 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-11-24T06:09:54.233 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:54.270 INFO:tasks.ceph.osd.5:Stopped 2023-11-24T06:09:54.271 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-11-24T06:09:54.271 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:54.348 INFO:tasks.ceph.osd.6:Stopped 2023-11-24T06:09:54.349 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-11-24T06:09:54.349 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:54.433 INFO:tasks.ceph.osd.7:Stopped 2023-11-24T06:09:54.434 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-24T06:09:54.434 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-11-24T06:09:54.434 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:54.458 INFO:tasks.ceph.mgr.y:Stopped 2023-11-24T06:09:54.459 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-24T06:09:54.459 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:54.482 INFO:tasks.ceph.mgr.x:Stopped 2023-11-24T06:09:54.483 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-24T06:09:54.483 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-24T06:09:54.483 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:54.509 INFO:tasks.ceph.mon.a:Stopped 2023-11-24T06:09:54.509 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-11-24T06:09:54.509 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:54.534 INFO:tasks.ceph.mon.b:Stopped 2023-11-24T06:09:54.534 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-11-24T06:09:54.535 INFO:teuthology.orchestra.run:waiting for 300 2023-11-24T06:09:54.586 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-11-24T06:09:54.586 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-11-24T06:09:54.586 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-11-24T06:09:54.586 INFO:tasks.ceph.mon.c:Stopped 2023-11-24T06:09:54.587 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-24T06:09:54.587 DEBUG:teuthology.orchestra.run.smithi091:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2023-11-24T06:09:54.620 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi091.front.sepia.ceph.com 2023-11-24T06:09:54.621 DEBUG:teuthology.orchestra.run.smithi091:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-24T06:09:54.938 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi091.front.sepia.ceph.com 2023-11-24T06:09:54.938 DEBUG:teuthology.orchestra.run.smithi091:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-24T06:09:55.037 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi091.front.sepia.ceph.com 2023-11-24T06:09:55.038 DEBUG:teuthology.orchestra.run.smithi091:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-24T06:09:55.118 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi091.front.sepia.ceph.com 2023-11-24T06:09:55.119 DEBUG:teuthology.orchestra.run.smithi091:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-11-24T06:09:55.206 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi094.front.sepia.ceph.com 2023-11-24T06:09:55.207 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-11-24T06:09:55.373 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi094.front.sepia.ceph.com 2023-11-24T06:09:55.374 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-11-24T06:09:55.468 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi094.front.sepia.ceph.com 2023-11-24T06:09:55.468 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-11-24T06:09:55.551 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi094.front.sepia.ceph.com 2023-11-24T06:09:55.552 DEBUG:teuthology.orchestra.run.smithi094:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-11-24T06:09:55.627 INFO:tasks.ceph:Archiving mon data... 2023-11-24T06:09:55.629 DEBUG:teuthology.misc:Transferring archived files from smithi091:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547/data/mon.a.tgz 2023-11-24T06:09:55.629 DEBUG:teuthology.orchestra.run.smithi091:> mktemp 2023-11-24T06:09:55.646 INFO:teuthology.orchestra.run.smithi091.stdout:/tmp/tmp.sP16OIOkIs 2023-11-24T06:09:55.647 DEBUG:teuthology.orchestra.run.smithi091:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.sP16OIOkIs 2023-11-24T06:09:55.772 DEBUG:teuthology.orchestra.run.smithi091:> sudo chmod 0666 /tmp/tmp.sP16OIOkIs 2023-11-24T06:09:55.858 DEBUG:teuthology.orchestra.remote:smithi091:/tmp/tmp.sP16OIOkIs is 318KB 2023-11-24T06:09:55.952 DEBUG:teuthology.orchestra.run.smithi091:> rm -fr /tmp/tmp.sP16OIOkIs 2023-11-24T06:09:55.968 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547/data/mon.b.tgz 2023-11-24T06:09:55.968 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2023-11-24T06:09:55.985 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.8JAGsAfPUp 2023-11-24T06:09:55.986 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.8JAGsAfPUp 2023-11-24T06:09:56.132 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.8JAGsAfPUp 2023-11-24T06:09:56.217 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.8JAGsAfPUp is 430KB 2023-11-24T06:09:56.313 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.8JAGsAfPUp 2023-11-24T06:09:56.329 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547/data/mon.c.tgz 2023-11-24T06:09:56.329 DEBUG:teuthology.orchestra.run.smithi094:> mktemp 2023-11-24T06:09:56.385 INFO:teuthology.orchestra.run.smithi094.stdout:/tmp/tmp.BvhGf4RCv1 2023-11-24T06:09:56.385 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.BvhGf4RCv1 2023-11-24T06:09:56.522 DEBUG:teuthology.orchestra.run.smithi094:> sudo chmod 0666 /tmp/tmp.BvhGf4RCv1 2023-11-24T06:09:56.608 DEBUG:teuthology.orchestra.remote:smithi094:/tmp/tmp.BvhGf4RCv1 is 430KB 2023-11-24T06:09:56.704 DEBUG:teuthology.orchestra.run.smithi094:> rm -fr /tmp/tmp.BvhGf4RCv1 2023-11-24T06:09:56.720 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-24T06:09:56.720 DEBUG:teuthology.orchestra.run.smithi091:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-24T06:09:56.723 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-24T06:09:56.786 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-24T06:09:56.786 DEBUG:teuthology.orchestra.run.smithi091:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-24T06:09:56.796 DEBUG:teuthology.orchestra.run.smithi094:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-24T06:09:56.840 INFO:teuthology.orchestra.run.smithi091.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-24T06:09:56.865 INFO:teuthology.orchestra.run.smithi094.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-24T06:09:56.870 INFO:tasks.ceph:Archiving crash dumps... 2023-11-24T06:09:56.872 DEBUG:teuthology.misc:Transferring archived files from smithi091:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547/remote/smithi091/crash 2023-11-24T06:09:56.873 DEBUG:teuthology.orchestra.run.smithi091:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-24T06:09:56.902 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547/remote/smithi094/crash 2023-11-24T06:09:56.903 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-24T06:09:56.936 INFO:tasks.ceph:Shutting down logrotate 2023-11-24T06:09:56.937 DEBUG:teuthology.orchestra.run.smithi091:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:09:56.946 DEBUG:teuthology.orchestra.run.smithi094:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-11-24T06:09:57.027 DEBUG:teuthology.orchestra.run.smithi091:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-11-24T06:09:57.055 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-11-24T06:09:57.081 INFO:tasks.ceph:Compressing logs... 2023-11-24T06:09:57.081 DEBUG:teuthology.orchestra.run.smithi091:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-24T06:09:57.098 DEBUG:teuthology.orchestra.run.smithi094:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-24T06:09:57.143 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149859.log 2023-11-24T06:09:57.143 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140007.log 2023-11-24T06:09:57.143 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.149859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148481.log 2023-11-24T06:09:57.143 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.140007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140910.log 2023-11-24T06:09:57.144 INFO:teuthology.orchestra.run.smithi091.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.149859.log.gz 2023-11-24T06:09:57.144 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.148481.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.140007.log.gz 2023-11-24T06:09:57.144 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144595.log 2023-11-24T06:09:57.144 INFO:teuthology.orchestra.run.smithi091.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.148481.log.gz 2023-11-24T06:09:57.144 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.140910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145898.log 2023-11-24T06:09:57.144 INFO:teuthology.orchestra.run.smithi091.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.140910.log.gz 2023-11-24T06:09:57.144 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.144595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151828.log 2023-11-24T06:09:57.144 INFO:teuthology.orchestra.run.smithi091.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.144595.log.gz 2023-11-24T06:09:57.145 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.145898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147101.log 2023-11-24T06:09:57.145 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.151828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150934.log 2023-11-24T06:09:57.145 INFO:teuthology.orchestra.run.smithi091.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.145898.log.gz 2023-11-24T06:09:57.145 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.147101.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.151828.log.gz 2023-11-24T06:09:57.145 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.148834.log 2023-11-24T06:09:57.145 INFO:teuthology.orchestra.run.smithi091.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.147101.log.gz 2023-11-24T06:09:57.145 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150934.log: /var/log/ceph/ceph-client.admin.147710.log 2023-11-24T06:09:57.146 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.0.148834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147894.log 2023-11-24T06:09:57.146 INFO:teuthology.orchestra.run.smithi091.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.148834.log.gz 2023-11-24T06:09:57.146 INFO:teuthology.orchestra.run.smithi091.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.150934.log.gz 2023-11-24T06:09:57.146 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.147710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148136.log 2023-11-24T06:09:57.146 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.147894.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.147710.log.gz 2023-11-24T06:09:57.146 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145361.log 2023-11-24T06:09:57.146 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.148136.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.147894.log.gz 2023-11-24T06:09:57.147 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147848.log 2023-11-24T06:09:57.147 INFO:teuthology.orchestra.run.smithi091.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.148136.log.gz 2023-11-24T06:09:57.147 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.145361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150135.log 2023-11-24T06:09:57.147 INFO:teuthology.orchestra.run.smithi091.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.145361.log.gz 2023-11-24T06:09:57.147 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141915.log 2023-11-24T06:09:57.147 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.147848.log: /var/log/ceph/ceph-client.admin.150135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140092.log 2023-11-24T06:09:57.147 INFO:teuthology.orchestra.run.smithi091.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.147848.log.gz 2023-11-24T06:09:57.147 INFO:teuthology.orchestra.run.smithi091.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.150135.log.gz 2023-11-24T06:09:57.148 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.141915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151782.log 2023-11-24T06:09:57.148 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.140092.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.141915.log.gz 2023-11-24T06:09:57.148 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145018.log 2023-11-24T06:09:57.148 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.151782.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.140092.log.gz 2023-11-24T06:09:57.148 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150880.log 2023-11-24T06:09:57.148 INFO:teuthology.orchestra.run.smithi091.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.151782.log.gz 2023-11-24T06:09:57.148 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.145018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140138.log 2023-11-24T06:09:57.148 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.150880.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.145018.log.gz 2023-11-24T06:09:57.149 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-11-24T06:09:57.149 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.140138.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.150880.log.gz 2023-11-24T06:09:57.149 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141556.log 2023-11-24T06:09:57.149 INFO:teuthology.orchestra.run.smithi091.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.140138.log.gz 2023-11-24T06:09:57.149 INFO:teuthology.orchestra.run.smithi091.stderr:gzip /var/log/ceph/ceph-osd.2.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.147009.log 2023-11-24T06:09:57.149 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.141556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142566.log 2023-11-24T06:09:57.149 INFO:teuthology.orchestra.run.smithi091.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.141556.log.gz 2023-11-24T06:09:57.149 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.147009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140323.log 2023-11-24T06:09:57.150 INFO:teuthology.orchestra.run.smithi091.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.147009.log.gz 2023-11-24T06:09:57.150 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.142566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.143478.log 2023-11-24T06:09:57.150 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.140323.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-24T06:09:57.150 INFO:teuthology.orchestra.run.smithi091.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.142566.log.gz 2023-11-24T06:09:57.150 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.0.143478.log: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.140323.log.gz-5 2023-11-24T06:09:57.150 INFO:teuthology.orchestra.run.smithi091.stderr:--verbose 0.0%-- /var/log/ceph/ceph-client.admin.142996.log -- replaced with /var/log/ceph/ceph-client.0.143478.log.gz 2023-11-24T06:09:57.150 INFO:teuthology.orchestra.run.smithi091.stderr: 2023-11-24T06:09:57.150 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149905.log 2023-11-24T06:09:57.151 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.142996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144780.log 2023-11-24T06:09:57.151 INFO:teuthology.orchestra.run.smithi091.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.142996.log.gz 2023-11-24T06:09:57.151 INFO:teuthology.orchestra.run.smithi091.stderr: 85.1%/var/log/ceph/ceph-client.admin.149905.log: -- replaced with /var/log/ceph/ceph.log.gz 2023-11-24T06:09:57.151 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142198.log 2023-11-24T06:09:57.151 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147618.log 2023-11-24T06:09:57.151 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.144780.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.149905.log.gz 2023-11-24T06:09:57.151 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.142198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142842.log 2023-11-24T06:09:57.152 INFO:teuthology.orchestra.run.smithi091.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.144780.log.gz 2023-11-24T06:09:57.152 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.147618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151394.log 2023-11-24T06:09:57.152 INFO:teuthology.orchestra.run.smithi091.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.142198.log.gz 2023-11-24T06:09:57.152 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.142842.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.147618.log.gz 2023-11-24T06:09:57.152 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150227.log 2023-11-24T06:09:57.152 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.151394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142750.log 2023-11-24T06:09:57.152 INFO:teuthology.orchestra.run.smithi091.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.142842.log.gz 2023-11-24T06:09:57.152 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.150227.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.151394.log.gz 2023-11-24T06:09:57.153 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144503.log 2023-11-24T06:09:57.153 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.142750.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.150227.log.gz 2023-11-24T06:09:57.153 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150273.log 2023-11-24T06:09:57.153 INFO:teuthology.orchestra.run.smithi091.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.142750.log.gz 2023-11-24T06:09:57.153 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.144503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146595.log 2023-11-24T06:09:57.153 INFO:teuthology.orchestra.run.smithi091.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.144503.log.gz 2023-11-24T06:09:57.153 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.150273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148940.log 2023-11-24T06:09:57.153 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.146595.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.150273.log.gz 2023-11-24T06:09:57.154 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148297.log 2023-11-24T06:09:57.154 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.148940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.144143.log 2023-11-24T06:09:57.154 INFO:teuthology.orchestra.run.smithi091.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.146595.log.gz 2023-11-24T06:09:57.154 INFO:teuthology.orchestra.run.smithi091.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.148940.log.gz 2023-11-24T06:09:57.154 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.148297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149702.log 2023-11-24T06:09:57.154 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140369.log 2023-11-24T06:09:57.154 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.2.144143.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.148297.log.gz 2023-11-24T06:09:57.155 INFO:teuthology.orchestra.run.smithi091.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.2.144143.log.gz 2023-11-24T06:09:57.155 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.149702.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-11-24T06:09:57.155 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.140369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150980.log 2023-11-24T06:09:57.155 INFO:teuthology.orchestra.run.smithi091.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.149702.log.gz 2023-11-24T06:09:57.155 INFO:teuthology.orchestra.run.smithi091.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.140369.log.gz 2023-11-24T06:09:57.155 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144411.log 2023-11-24T06:09:57.155 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.150980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146411.log 2023-11-24T06:09:57.155 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.144411.log: gzip -5 88.1%--verbose -- /var/log/ceph/ceph-client.admin.149032.log -- replaced with /var/log/ceph/ceph-client.admin.150980.log.gz 2023-11-24T06:09:57.156 INFO:teuthology.orchestra.run.smithi091.stderr: 2023-11-24T06:09:57.156 INFO:teuthology.orchestra.run.smithi091.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.144411.log.gz 2023-11-24T06:09:57.156 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.146411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143708.log 2023-11-24T06:09:57.156 INFO:teuthology.orchestra.run.smithi091.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.146411.log.gz 2023-11-24T06:09:57.156 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142474.log 2023-11-24T06:09:57.156 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.149032.log: /var/log/ceph/ceph-client.admin.143708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145085.log 2023-11-24T06:09:57.156 INFO:teuthology.orchestra.run.smithi091.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.149032.log.gz 2023-11-24T06:09:57.156 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.142474.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.143708.log.gz 2023-11-24T06:09:57.157 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146185.log 2023-11-24T06:09:57.157 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142796.log 2023-11-24T06:09:57.157 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.145085.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.142474.log.gz 2023-11-24T06:09:57.157 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.146185.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.145085.log.gz 2023-11-24T06:09:57.157 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147942.log 2023-11-24T06:09:57.157 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-11-24T06:09:57.157 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.142796.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.146185.log.gz 2023-11-24T06:09:57.158 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.147942.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.142796.log.gz 2023-11-24T06:09:57.158 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146917.log 2023-11-24T06:09:57.158 INFO:teuthology.orchestra.run.smithi091.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.147942.log.gz 2023-11-24T06:09:57.158 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145453.log 2023-11-24T06:09:57.158 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.146917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141209.log 2023-11-24T06:09:57.158 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.145453.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.146917.log.gz 2023-11-24T06:09:57.158 INFO:teuthology.orchestra.run.smithi091.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.145453.log.gz 2023-11-24T06:09:57.159 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.141209.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.141209.log.gz 2023-11-24T06:09:57.159 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144733.log 2023-11-24T06:09:57.159 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151072.log 2023-11-24T06:09:57.159 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.144733.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.144733.log.gz 2023-11-24T06:09:57.159 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.151072.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.151072.log.gz 2023-11-24T06:09:57.159 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.144139.log 2023-11-24T06:09:57.159 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151738.log 2023-11-24T06:09:57.159 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.2.144139.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-11-24T06:09:57.160 INFO:teuthology.orchestra.run.smithi091.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.2.144139.log.gz 2023-11-24T06:09:57.160 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.151738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149748.log 2023-11-24T06:09:57.160 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-osd.3.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.151738.log.gz 2023-11-24T06:09:57.160 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.149748.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.149748.log.gz 2023-11-24T06:09:57.160 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147664.log 2023-11-24T06:09:57.160 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146036.log 2023-11-24T06:09:57.160 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.147664.log: /var/log/ceph/ceph-client.admin.146036.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.147664.log.gz 2023-11-24T06:09:57.160 INFO:teuthology.orchestra.run.smithi091.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.146036.log.gz 2023-11-24T06:09:57.161 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.148830.log 2023-11-24T06:09:57.161 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150788.log 2023-11-24T06:09:57.174 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.140070.log 2023-11-24T06:09:57.174 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-11-24T06:09:57.174 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.1.140070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133545.log 2023-11-24T06:09:57.174 INFO:teuthology.orchestra.run.smithi094.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.140070.log.gz 2023-11-24T06:09:57.174 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.141459.log 2023-11-24T06:09:57.174 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-24T06:09:57.175 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.133545.log: /var/log/ceph/ceph-client.1.141459.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-11-24T06:09:57.175 INFO:teuthology.orchestra.run.smithi094.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.1.141459.log.gz 2023-11-24T06:09:57.175 INFO:teuthology.orchestra.run.smithi094.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.133545.log.gz 2023-11-24T06:09:57.175 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133684.log 2023-11-24T06:09:57.175 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133637.log 2023-11-24T06:09:57.175 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.133684.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-11-24T06:09:57.175 INFO:teuthology.orchestra.run.smithi094.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.133684.log.gz 2023-11-24T06:09:57.176 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.133637.log: 91.6% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-24T06:09:57.176 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-mds.d.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.133637.log.gz 2023-11-24T06:09:57.176 INFO:teuthology.orchestra.run.smithi094.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-11-24T06:09:57.176 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133592.log 2023-11-24T06:09:57.176 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-24T06:09:57.176 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.133592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133774.log 2023-11-24T06:09:57.176 INFO:teuthology.orchestra.run.smithi094.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.133592.log.gz 2023-11-24T06:09:57.176 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.140685.log 2023-11-24T06:09:57.177 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-11-24T06:09:57.177 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.133774.log: /var/log/ceph/ceph-client.3.140685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.140689.log 2023-11-24T06:09:57.177 INFO:teuthology.orchestra.run.smithi094.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.3.140685.log.gz 2023-11-24T06:09:57.177 INFO:teuthology.orchestra.run.smithi094.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.133774.log.gz 2023-11-24T06:09:57.177 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-24T06:09:57.177 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.3.140689.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-11-24T06:09:57.177 INFO:teuthology.orchestra.run.smithi094.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.3.140689.log.gz 2023-11-24T06:09:57.178 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-11-24T06:09:57.178 INFO:teuthology.orchestra.run.smithi094.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-24T06:09:57.178 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.140078.log 2023-11-24T06:09:57.178 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-client.1.140078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.140078.log.gz 2023-11-24T06:09:57.178 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-11-24T06:09:57.178 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-mon.b.log: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-24T06:09:57.178 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133729.log 2023-11-24T06:09:57.178 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.133729.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.133729.log.gz 2023-11-24T06:09:57.179 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.141455.log 2023-11-24T06:09:57.179 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.1.141455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.141455.log.gz 2023-11-24T06:09:57.179 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-11-24T06:09:57.179 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-mds.b.log: 91.6% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-11-24T06:09:57.179 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133819.log 2023-11-24T06:09:57.179 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.133819.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.133819.log.gz 2023-11-24T06:09:57.179 INFO:teuthology.orchestra.run.smithi094.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133872.log 2023-11-24T06:09:57.180 INFO:teuthology.orchestra.run.smithi094.stderr:/var/log/ceph/ceph-client.admin.133872.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.133872.log.gz 2023-11-24T06:09:57.197 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.0.148830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.148830.log.gz 2023-11-24T06:09:57.197 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.150788.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.150788.log.gz 2023-11-24T06:09:57.197 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142704.log 2023-11-24T06:09:57.197 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145672.log 2023-11-24T06:09:57.198 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.142704.log: /var/log/ceph/ceph-client.admin.145672.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.142704.log.gz 2023-11-24T06:09:57.198 INFO:teuthology.orchestra.run.smithi091.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.145672.log.gz 2023-11-24T06:09:57.198 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142658.log 2023-11-24T06:09:57.198 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141207.log 2023-11-24T06:09:57.198 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.142658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151348.log 2023-11-24T06:09:57.198 INFO:teuthology.orchestra.run.smithi091.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.142658.log.gz 2023-11-24T06:09:57.198 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.141207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146687.log 2023-11-24T06:09:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.151348.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.141207.log.gz 2023-11-24T06:09:57.199 INFO:teuthology.orchestra.run.smithi091.stderr: 87.3%/var/log/ceph/ceph-client.admin.146687.log: -- replaced with /var/log/ceph/ceph-client.admin.151348.log.gz 2023-11-24T06:09:57.199 INFO:teuthology.orchestra.run.smithi091.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.146687.log.gz 2023-11-24T06:09:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145407.log 2023-11-24T06:09:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149997.log 2023-11-24T06:09:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.145407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146139.log 2023-11-24T06:09:57.199 INFO:teuthology.orchestra.run.smithi091.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.145407.log.gz 2023-11-24T06:09:57.199 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.149997.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.149997.log.gz 2023-11-24T06:09:57.200 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.146139.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.146139.log.gz 2023-11-24T06:09:57.200 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141574.log 2023-11-24T06:09:57.200 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149124.log 2023-11-24T06:09:57.200 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.141574.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.141574.log.gz 2023-11-24T06:09:57.200 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.149124.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.149124.log.gz 2023-11-24T06:09:57.200 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141946.log 2023-11-24T06:09:57.200 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151486.log 2023-11-24T06:09:57.201 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.141946.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.141946.log.gz 2023-11-24T06:09:57.201 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.151486.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.151486.log.gz 2023-11-24T06:09:57.201 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141203.log 2023-11-24T06:09:57.201 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144549.log 2023-11-24T06:09:57.201 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.141203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145944.log 2023-11-24T06:09:57.201 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.144549.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.141203.log.gz 2023-11-24T06:09:57.201 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.145944.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.144549.log.gz 2023-11-24T06:09:57.202 INFO:teuthology.orchestra.run.smithi091.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.145944.log.gz 2023-11-24T06:09:57.202 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-11-24T06:09:57.202 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141577.log 2023-11-24T06:09:57.202 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.141577.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.141577.log.gz 2023-11-24T06:09:57.202 INFO:teuthology.orchestra.run.smithi091.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-11-24T06:09:57.202 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148184.log 2023-11-24T06:09:57.202 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141922.log 2023-11-24T06:09:57.203 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.148184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142290.log 2023-11-24T06:09:57.203 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.141922.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.148184.log.gz 2023-11-24T06:09:57.203 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.142290.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.141922.log.gz 2023-11-24T06:09:57.203 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141194.log 2023-11-24T06:09:57.203 INFO:teuthology.orchestra.run.smithi091.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.142290.log.gz 2023-11-24T06:09:57.203 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148986.log 2023-11-24T06:09:57.203 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.141194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149656.log 2023-11-24T06:09:57.203 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.148986.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.141194.log.gz 2023-11-24T06:09:57.204 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.149656.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.148986.log.gz 2023-11-24T06:09:57.204 INFO:teuthology.orchestra.run.smithi091.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.149656.log.gz 2023-11-24T06:09:57.204 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141916.log 2023-11-24T06:09:57.204 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142382.log 2023-11-24T06:09:57.204 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.141916.log: /var/log/ceph/ceph-client.admin.142382.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.141916.log.gz 2023-11-24T06:09:57.204 INFO:teuthology.orchestra.run.smithi091.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.142382.log.gz 2023-11-24T06:09:57.204 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148527.log 2023-11-24T06:09:57.204 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145315.log 2023-11-24T06:09:57.205 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.148527.log: /var/log/ceph/ceph-client.admin.145315.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.148527.log.gz 2023-11-24T06:09:57.205 INFO:teuthology.orchestra.run.smithi091.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.145315.log.gz 2023-11-24T06:09:57.205 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140726.log 2023-11-24T06:09:57.205 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149338.log 2023-11-24T06:09:57.205 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.140726.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.140726.log.gz 2023-11-24T06:09:57.205 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.149338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150384.log 2023-11-24T06:09:57.205 INFO:teuthology.orchestra.run.smithi091.stderr:gzip 88.3%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.149338.log.gz-- 2023-11-24T06:09:57.206 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.146231.log 2023-11-24T06:09:57.206 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.150384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141568.log 2023-11-24T06:09:57.206 INFO:teuthology.orchestra.run.smithi091.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.150384.log.gz 2023-11-24T06:09:57.206 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.146231.log: /var/log/ceph/ceph-client.admin.141568.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.146231.log.gz 2023-11-24T06:09:57.206 INFO:teuthology.orchestra.run.smithi091.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.141568.log.gz 2023-11-24T06:09:57.206 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142152.log 2023-11-24T06:09:57.206 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146503.log 2023-11-24T06:09:57.207 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.142152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139898.log 2023-11-24T06:09:57.207 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.146503.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.142152.log.gz 2023-11-24T06:09:57.207 INFO:teuthology.orchestra.run.smithi091.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.146503.log.gz 2023-11-24T06:09:57.207 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.139898.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.139898.log.gz 2023-11-24T06:09:57.207 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141195.log 2023-11-24T06:09:57.207 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150834.log 2023-11-24T06:09:57.207 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.141195.log: /var/log/ceph/ceph-client.admin.150834.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.141195.log.gz 2023-11-24T06:09:57.207 INFO:teuthology.orchestra.run.smithi091.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.150834.log.gz 2023-11-24T06:09:57.208 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139802.log 2023-11-24T06:09:57.208 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-24T06:09:57.208 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.139802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145990.log 2023-11-24T06:09:57.208 INFO:teuthology.orchestra.run.smithi091.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.139802.log.gz 2023-11-24T06:09:57.208 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.145990.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.145990.log.gz 2023-11-24T06:09:57.208 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151026.log 2023-11-24T06:09:57.208 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149292.log 2023-11-24T06:09:57.208 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.151026.log: 88.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-24T06:09:57.209 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.149292.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.151026.log.gz 2023-11-24T06:09:57.209 INFO:teuthology.orchestra.run.smithi091.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.149292.log.gz 2023-11-24T06:09:57.209 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140231.log 2023-11-24T06:09:57.209 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145580.log 2023-11-24T06:09:57.209 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.140231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140864.log 2023-11-24T06:09:57.209 INFO:teuthology.orchestra.run.smithi091.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.140231.log.gz 2023-11-24T06:09:57.209 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.145580.log: /var/log/ceph/ceph-client.admin.140864.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.145580.log.gz 2023-11-24T06:09:57.210 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145718.log 2023-11-24T06:09:57.210 INFO:teuthology.orchestra.run.smithi091.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.140864.log.gz 2023-11-24T06:09:57.210 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.145718.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.145718.log.gz 2023-11-24T06:09:57.210 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151645.log 2023-11-24T06:09:57.210 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150043.log 2023-11-24T06:09:57.210 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.151645.log: /var/log/ceph/ceph-client.admin.150043.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.151645.log.gz 2023-11-24T06:09:57.210 INFO:teuthology.orchestra.run.smithi091.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.150043.log.gz 2023-11-24T06:09:57.211 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147258.log 2023-11-24T06:09:57.211 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144365.log 2023-11-24T06:09:57.211 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.147258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146825.log 2023-11-24T06:09:57.211 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.144365.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.147258.log.gz 2023-11-24T06:09:57.211 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.146825.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.144365.log.gz 2023-11-24T06:09:57.211 INFO:teuthology.orchestra.run.smithi091.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.146825.log.gz 2023-11-24T06:09:57.211 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144641.log 2023-11-24T06:09:57.211 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140956.log 2023-11-24T06:09:57.212 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.144641.log: /var/log/ceph/ceph-client.admin.140956.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.144641.log.gz 2023-11-24T06:09:57.212 INFO:teuthology.orchestra.run.smithi091.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.140956.log.gz 2023-11-24T06:09:57.212 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141784.log 2023-11-24T06:09:57.212 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148597.log 2023-11-24T06:09:57.212 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.141784.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.141784.log.gz 2023-11-24T06:09:57.212 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.148597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140277.log 2023-11-24T06:09:57.212 INFO:teuthology.orchestra.run.smithi091.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.148597.log.gz 2023-11-24T06:09:57.212 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145626.log 2023-11-24T06:09:57.213 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.140277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141921.log 2023-11-24T06:09:57.213 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.145626.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.140277.log.gz 2023-11-24T06:09:57.213 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.141921.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.145626.log.gz 2023-11-24T06:09:57.213 INFO:teuthology.orchestra.run.smithi091.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.141921.log.gz 2023-11-24T06:09:57.213 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141570.log 2023-11-24T06:09:57.213 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147802.log 2023-11-24T06:09:57.213 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.141570.log: /var/log/ceph/ceph-client.admin.147802.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.141570.log.gz 2023-11-24T06:09:57.214 INFO:teuthology.orchestra.run.smithi091.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.147802.log.gz 2023-11-24T06:09:57.214 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142336.log 2023-11-24T06:09:57.214 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133436.log 2023-11-24T06:09:57.214 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.142336.log: /var/log/ceph/ceph-client.admin.133436.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.142336.log.gz 2023-11-24T06:09:57.214 INFO:teuthology.orchestra.run.smithi091.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.133436.log.gz 2023-11-24T06:09:57.214 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151210.log 2023-11-24T06:09:57.214 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133746.log 2023-11-24T06:09:57.215 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.151210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150522.log 2023-11-24T06:09:57.215 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.133746.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.151210.log.gz 2023-11-24T06:09:57.215 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.150522.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.133746.log.gz 2023-11-24T06:09:57.215 INFO:teuthology.orchestra.run.smithi091.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.150522.log.gz 2023-11-24T06:09:57.215 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145131.log 2023-11-24T06:09:57.215 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142520.log 2023-11-24T06:09:57.215 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.145131.log: /var/log/ceph/ceph-client.admin.142520.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.145131.log.gz 2023-11-24T06:09:57.215 INFO:teuthology.orchestra.run.smithi091.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.142520.log.gz 2023-11-24T06:09:57.216 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142428.log 2023-11-24T06:09:57.216 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147572.log 2023-11-24T06:09:57.216 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.142428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148090.log 2023-11-24T06:09:57.216 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.147572.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.142428.log.gz 2023-11-24T06:09:57.216 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.148090.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.147572.log.gz 2023-11-24T06:09:57.216 INFO:teuthology.orchestra.run.smithi091.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.148090.log.gz 2023-11-24T06:09:57.216 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151440.log 2023-11-24T06:09:57.217 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149430.log 2023-11-24T06:09:57.217 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.151440.log: /var/log/ceph/ceph-client.admin.149430.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.151440.log.gz 2023-11-24T06:09:57.217 INFO:teuthology.orchestra.run.smithi091.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.149430.log.gz 2023-11-24T06:09:57.217 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148343.log 2023-11-24T06:09:57.217 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-24T06:09:57.217 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.148343.log: /var/log/ceph/ceph-osd.1.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.148343.log.gz 2023-11-24T06:09:57.217 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140417.log 2023-11-24T06:09:57.217 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150337.log 2023-11-24T06:09:57.218 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.140417.log: /var/log/ceph/ceph-client.admin.150337.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.140417.log.gz 2023-11-24T06:09:57.218 INFO:teuthology.orchestra.run.smithi091.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.150337.log.gz 2023-11-24T06:09:57.218 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148435.log 2023-11-24T06:09:57.218 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147434.log 2023-11-24T06:09:57.218 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.148435.log: /var/log/ceph/ceph-client.admin.147434.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.148435.log.gz 2023-11-24T06:09:57.218 INFO:teuthology.orchestra.run.smithi091.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.147434.log.gz 2023-11-24T06:09:57.218 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146779.log 2023-11-24T06:09:57.219 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149078.log 2023-11-24T06:09:57.219 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.146779.log: 88.1%/var/log/ceph/ceph-client.admin.149078.log: -- replaced with /var/log/ceph/ceph-client.admin.146779.log.gz 2023-11-24T06:09:57.219 INFO:teuthology.orchestra.run.smithi091.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.149078.log.gz 2023-11-24T06:09:57.219 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140540.log 2023-11-24T06:09:57.219 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147212.log 2023-11-24T06:09:57.219 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.140540.log: /var/log/ceph/ceph-client.admin.147212.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.140540.log.gz 2023-11-24T06:09:57.219 INFO:teuthology.orchestra.run.smithi091.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.147212.log.gz 2023-11-24T06:09:57.219 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140185.log 2023-11-24T06:09:57.220 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141202.log 2023-11-24T06:09:57.220 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.140185.log: /var/log/ceph/ceph-client.admin.141202.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.140185.log.gz 2023-11-24T06:09:57.220 INFO:teuthology.orchestra.run.smithi091.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.141202.log.gz 2023-11-24T06:09:57.220 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141205.log 2023-11-24T06:09:57.220 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146323.log 2023-11-24T06:09:57.220 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.141205.log: /var/log/ceph/ceph-client.admin.146323.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.141205.log.gz 2023-11-24T06:09:57.220 INFO:teuthology.orchestra.run.smithi091.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.146323.log.gz 2023-11-24T06:09:57.220 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146871.log 2023-11-24T06:09:57.221 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149184.log 2023-11-24T06:09:57.221 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.146871.log: /var/log/ceph/ceph-client.admin.149184.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.146871.log.gz 2023-11-24T06:09:57.221 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147165.log 2023-11-24T06:09:57.221 INFO:teuthology.orchestra.run.smithi091.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.149184.log.gz 2023-11-24T06:09:57.221 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147756.log 2023-11-24T06:09:57.221 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.147165.log: /var/log/ceph/ceph-client.admin.147756.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.147165.log.gz 2023-11-24T06:09:57.221 INFO:teuthology.orchestra.run.smithi091.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.147756.log.gz 2023-11-24T06:09:57.222 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140818.log 2023-11-24T06:09:57.222 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140586.log 2023-11-24T06:09:57.222 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.140818.log: /var/log/ceph/ceph-client.admin.140586.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.140818.log.gz 2023-11-24T06:09:57.222 INFO:teuthology.orchestra.run.smithi091.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.140586.log.gz 2023-11-24T06:09:57.222 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148044.log 2023-11-24T06:09:57.222 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140680.log 2023-11-24T06:09:57.236 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.148044.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.148044.log.gz/var/log/ceph/ceph-client.admin.140680.log: 2023-11-24T06:09:57.236 INFO:teuthology.orchestra.run.smithi091.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.140680.log.gz 2023-11-24T06:09:57.236 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149951.log 2023-11-24T06:09:57.237 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145852.log 2023-11-24T06:09:57.237 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.149951.log: /var/log/ceph/ceph-client.admin.145852.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.149951.log.gz 2023-11-24T06:09:57.237 INFO:teuthology.orchestra.run.smithi091.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.145852.log.gz 2023-11-24T06:09:57.237 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149476.log 2023-11-24T06:09:57.237 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146641.log 2023-11-24T06:09:57.237 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.149476.log: /var/log/ceph/ceph-client.admin.146641.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.149476.log.gz 2023-11-24T06:09:57.237 INFO:teuthology.orchestra.run.smithi091.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.146641.log.gz 2023-11-24T06:09:57.238 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146733.log 2023-11-24T06:09:57.238 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144872.log 2023-11-24T06:09:57.238 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.146733.log: /var/log/ceph/ceph-client.admin.144872.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.146733.log.gz 2023-11-24T06:09:57.238 INFO:teuthology.orchestra.run.smithi091.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.144872.log.gz 2023-11-24T06:09:57.238 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144687.log 2023-11-24T06:09:57.238 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147526.log 2023-11-24T06:09:57.238 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.144687.log: /var/log/ceph/ceph-client.admin.147526.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.144687.log.gz 2023-11-24T06:09:57.239 INFO:teuthology.orchestra.run.smithi091.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.147526.log.gz 2023-11-24T06:09:57.239 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145223.log 2023-11-24T06:09:57.239 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151302.log 2023-11-24T06:09:57.239 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.145223.log: /var/log/ceph/ceph-client.admin.151302.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.145223.log.gz 2023-11-24T06:09:57.239 INFO:teuthology.orchestra.run.smithi091.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.151302.log.gz 2023-11-24T06:09:57.239 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142244.log 2023-11-24T06:09:57.239 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142612.log 2023-11-24T06:09:57.239 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.142244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133877.log 2023-11-24T06:09:57.240 INFO:teuthology.orchestra.run.smithi091.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.142244.log.gz 2023-11-24T06:09:57.240 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.142612.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.142612.log.gz 2023-11-24T06:09:57.240 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149384.log 2023-11-24T06:09:57.240 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.133877.log: /var/log/ceph/ceph-client.admin.149384.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.133877.log.gz 2023-11-24T06:09:57.240 INFO:teuthology.orchestra.run.smithi091.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.149384.log.gz 2023-11-24T06:09:57.240 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140632.log 2023-11-24T06:09:57.240 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141569.log 2023-11-24T06:09:57.241 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.140632.log: /var/log/ceph/ceph-client.admin.141569.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.140632.log.gz 2023-11-24T06:09:57.241 INFO:teuthology.orchestra.run.smithi091.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.141569.log.gz 2023-11-24T06:09:57.241 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.127145.log 2023-11-24T06:09:57.241 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148251.log 2023-11-24T06:09:57.241 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph.tmp-client.admin.127145.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.127145.log.gz 2023-11-24T06:09:57.241 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.148251.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.148251.log.gz 2023-11-24T06:09:57.241 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140772.log 2023-11-24T06:09:57.241 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147480.log 2023-11-24T06:09:57.242 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.140772.log: /var/log/ceph/ceph-client.admin.147480.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.140772.log.gz 2023-11-24T06:09:57.242 INFO:teuthology.orchestra.run.smithi091.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.147480.log.gz 2023-11-24T06:09:57.242 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151118.log 2023-11-24T06:09:57.242 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147348.log 2023-11-24T06:09:57.242 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.151118.log: /var/log/ceph/ceph-client.admin.147348.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.151118.log.gz 2023-11-24T06:09:57.242 INFO:teuthology.orchestra.run.smithi091.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.147348.log.gz 2023-11-24T06:09:57.242 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149813.log 2023-11-24T06:09:57.243 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141573.log 2023-11-24T06:09:57.243 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.149813.log: /var/log/ceph/ceph-client.admin.141573.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.149813.log.gz 2023-11-24T06:09:57.243 INFO:teuthology.orchestra.run.smithi091.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.141573.log.gz 2023-11-24T06:09:57.243 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141208.log 2023-11-24T06:09:57.243 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150089.log 2023-11-24T06:09:57.243 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.141208.log: /var/log/ceph/ceph-client.admin.150089.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.141208.log.gz 2023-11-24T06:09:57.243 INFO:teuthology.orchestra.run.smithi091.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.150089.log.gz 2023-11-24T06:09:57.243 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151599.log 2023-11-24T06:09:57.244 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151532.log 2023-11-24T06:09:57.244 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.151599.log: /var/log/ceph/ceph-client.admin.151532.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.151599.log.gz 2023-11-24T06:09:57.244 INFO:teuthology.orchestra.run.smithi091.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.151532.log.gz 2023-11-24T06:09:57.244 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150742.log 2023-11-24T06:09:57.244 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141572.log 2023-11-24T06:09:57.244 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.150742.log: /var/log/ceph/ceph-client.admin.141572.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.150742.log.gz 2023-11-24T06:09:57.244 INFO:teuthology.orchestra.run.smithi091.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.141572.log.gz 2023-11-24T06:09:57.245 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145764.log 2023-11-24T06:09:57.245 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141918.log 2023-11-24T06:09:57.245 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.145764.log: /var/log/ceph/ceph-client.admin.141918.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.145764.log.gz 2023-11-24T06:09:57.245 INFO:teuthology.orchestra.run.smithi091.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.141918.log.gz 2023-11-24T06:09:57.245 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146277.log 2023-11-24T06:09:57.245 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.143486.log 2023-11-24T06:09:57.245 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.146277.log: /var/log/ceph/ceph-client.0.143486.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.146277.log.gz 2023-11-24T06:09:57.245 INFO:teuthology.orchestra.run.smithi091.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.143486.log.gz 2023-11-24T06:09:57.246 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145177.log 2023-11-24T06:09:57.246 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141002.log 2023-11-24T06:09:57.246 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.145177.log: /var/log/ceph/ceph-client.admin.141002.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.145177.log.gz 2023-11-24T06:09:57.246 INFO:teuthology.orchestra.run.smithi091.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.141002.log.gz 2023-11-24T06:09:57.246 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144457.log 2023-11-24T06:09:57.246 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141920.log 2023-11-24T06:09:57.246 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.144457.log: /var/log/ceph/ceph-client.admin.141920.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.144457.log.gz 2023-11-24T06:09:57.247 INFO:teuthology.orchestra.run.smithi091.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.141920.log.gz 2023-11-24T06:09:57.247 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149564.log 2023-11-24T06:09:57.247 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147990.log 2023-11-24T06:09:57.247 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.149564.log: /var/log/ceph/ceph-client.admin.147990.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.149564.log.gz 2023-11-24T06:09:57.247 INFO:teuthology.orchestra.run.smithi091.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.147990.log.gz 2023-11-24T06:09:57.247 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151256.log 2023-11-24T06:09:57.247 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150181.log 2023-11-24T06:09:57.247 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.151256.log: /var/log/ceph/ceph-client.admin.150181.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.151256.log.gz 2023-11-24T06:09:57.248 INFO:teuthology.orchestra.run.smithi091.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.150181.log.gz 2023-11-24T06:09:57.248 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146457.log 2023-11-24T06:09:57.248 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144926.log 2023-11-24T06:09:57.248 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.146457.log: /var/log/ceph/ceph-client.admin.144926.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.146457.log.gz 2023-11-24T06:09:57.248 INFO:teuthology.orchestra.run.smithi091.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.144926.log.gz 2023-11-24T06:09:57.248 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151691.log 2023-11-24T06:09:57.248 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-24T06:09:57.248 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.151691.log: /var/log/ceph/ceph-mon.a.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.151691.log.gz 2023-11-24T06:09:57.249 INFO:teuthology.orchestra.run.smithi091.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-24T06:09:57.249 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140465.log 2023-11-24T06:09:57.249 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144826.log 2023-11-24T06:09:57.249 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.140465.log: /var/log/ceph/ceph-client.admin.144826.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.140465.log.gz 2023-11-24T06:09:57.249 INFO:teuthology.orchestra.run.smithi091.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.144826.log.gz 2023-11-24T06:09:57.249 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146963.log 2023-11-24T06:09:57.249 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150476.log 2023-11-24T06:09:57.250 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.146963.log: /var/log/ceph/ceph-client.admin.150476.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.146963.log.gz 2023-11-24T06:09:57.250 INFO:teuthology.orchestra.run.smithi091.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.150476.log.gz 2023-11-24T06:09:57.250 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145269.log 2023-11-24T06:09:57.250 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147055.log 2023-11-24T06:09:57.250 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.145269.log: /var/log/ceph/ceph-client.admin.147055.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.145269.log.gz 2023-11-24T06:09:57.250 INFO:teuthology.orchestra.run.smithi091.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.147055.log.gz 2023-11-24T06:09:57.250 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150430.log 2023-11-24T06:09:57.251 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146549.log 2023-11-24T06:09:57.251 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.150430.log: /var/log/ceph/ceph-client.admin.146549.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.150430.log.gz 2023-11-24T06:09:57.251 INFO:teuthology.orchestra.run.smithi091.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.146549.log.gz 2023-11-24T06:09:57.251 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151164.log 2023-11-24T06:09:57.251 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-11-24T06:09:57.251 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.151164.log: /var/log/ceph/ceph-mgr.y.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.151164.log.gz 2023-11-24T06:09:57.252 INFO:teuthology.orchestra.run.smithi094.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-11-24T06:09:57.252 INFO:teuthology.orchestra.run.smithi094.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-11-24T06:09:57.253 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144972.log 2023-11-24T06:09:57.253 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.144972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148389.log 2023-11-24T06:09:57.253 INFO:teuthology.orchestra.run.smithi091.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.144972.log.gz 2023-11-24T06:09:57.253 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.148389.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.148389.log.gz 2023-11-24T06:09:57.254 INFO:teuthology.orchestra.run.smithi091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149610.log 2023-11-24T06:09:57.254 INFO:teuthology.orchestra.run.smithi091.stderr:/var/log/ceph/ceph-client.admin.149610.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.149610.log.gz 2023-11-24T06:09:57.254 INFO:teuthology.orchestra.run.smithi091.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-24T06:09:57.257 INFO:teuthology.orchestra.run.smithi094.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-11-24T06:09:57.259 INFO:teuthology.orchestra.run.smithi091.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 94.5% 2023-11-24T06:09:57.259 INFO:teuthology.orchestra.run.smithi091.stderr: -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-11-24T06:09:57.272 INFO:teuthology.orchestra.run.smithi094.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-11-24T06:09:57.276 INFO:teuthology.orchestra.run.smithi091.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-11-24T06:09:57.280 INFO:teuthology.orchestra.run.smithi094.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-11-24T06:09:57.281 INFO:teuthology.orchestra.run.smithi094.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-11-24T06:09:57.282 INFO:teuthology.orchestra.run.smithi094.stderr: 2023-11-24T06:09:57.282 INFO:teuthology.orchestra.run.smithi094.stderr:real 0m0.139s 2023-11-24T06:09:57.283 INFO:teuthology.orchestra.run.smithi094.stderr:user 0m0.627s 2023-11-24T06:09:57.283 INFO:teuthology.orchestra.run.smithi094.stderr:sys 0m0.040s 2023-11-24T06:09:57.300 INFO:teuthology.orchestra.run.smithi091.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-24T06:09:57.400 INFO:teuthology.orchestra.run.smithi091.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-24T06:09:57.402 INFO:teuthology.orchestra.run.smithi091.stderr: 2023-11-24T06:09:57.402 INFO:teuthology.orchestra.run.smithi091.stderr:real 0m0.290s 2023-11-24T06:09:57.402 INFO:teuthology.orchestra.run.smithi091.stderr:user 0m0.854s 2023-11-24T06:09:57.402 INFO:teuthology.orchestra.run.smithi091.stderr:sys 0m0.150s 2023-11-24T06:09:57.402 INFO:tasks.ceph:Archiving logs... 2023-11-24T06:09:57.403 DEBUG:teuthology.misc:Transferring archived files from smithi091:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547/remote/smithi091/log 2023-11-24T06:09:57.404 DEBUG:teuthology.orchestra.run.smithi091:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-24T06:09:57.729 DEBUG:teuthology.misc:Transferring archived files from smithi094:/var/log/ceph to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547/remote/smithi094/log 2023-11-24T06:09:57.729 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-24T06:09:57.850 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-24T06:09:57.870 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-24T06:09:57.870 DEBUG:teuthology.orchestra.run.smithi091:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-24T06:09:57.873 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-24T06:09:57.948 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-24T06:09:57.957 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-24T06:09:57.958 DEBUG:teuthology.orchestra.run.smithi091:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-24T06:09:57.962 DEBUG:teuthology.orchestra.run.smithi094:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-24T06:09:57.975 INFO:teuthology.orchestra.run.smithi091.stderr:bash: ntpq: command not found 2023-11-24T06:09:57.977 INFO:teuthology.orchestra.run.smithi094.stderr:bash: ntpq: command not found 2023-11-24T06:09:57.979 INFO:teuthology.orchestra.run.smithi091.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-24T06:09:57.979 INFO:teuthology.orchestra.run.smithi091.stdout:=============================================================================== 2023-11-24T06:09:57.979 INFO:teuthology.orchestra.run.smithi091.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 1 +2167us[+2167us] +/- 67ms 2023-11-24T06:09:57.979 INFO:teuthology.orchestra.run.smithi091.stdout:^* hv02.front.sepia.ceph.com 3 6 377 2 -1654us[-1656us] +/- 55ms 2023-11-24T06:09:57.980 INFO:teuthology.orchestra.run.smithi091.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 -180us[ -180us] +/- 87ms 2023-11-24T06:09:57.980 INFO:teuthology.orchestra.run.smithi091.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:09:57.982 INFO:teuthology.orchestra.run.smithi094.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-24T06:09:57.982 INFO:teuthology.orchestra.run.smithi094.stdout:=============================================================================== 2023-11-24T06:09:57.982 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 1 +2178us[+2181us] +/- 67ms 2023-11-24T06:09:57.983 INFO:teuthology.orchestra.run.smithi094.stdout:^* hv02.front.sepia.ceph.com 3 6 377 1 -1647us[-1644us] +/- 55ms 2023-11-24T06:09:57.983 INFO:teuthology.orchestra.run.smithi094.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 -173us[ -170us] +/- 87ms 2023-11-24T06:09:57.983 INFO:teuthology.orchestra.run.smithi094.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-24T06:09:57.983 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-24T06:09:57.992 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-24T06:09:57.993 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-24T06:09:58.002 DEBUG:teuthology.orchestra.run.smithi091:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-24T06:09:58.117 DEBUG:teuthology.orchestra.run.smithi094:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-24T06:09:58.209 DEBUG:teuthology.orchestra.run.smithi091:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-24T06:09:58.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:09:58.240 DEBUG:teuthology.orchestra.run.smithi094:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-24T06:09:58.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:09:58.272 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-24T06:09:58.299 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-24T06:09:58.311 INFO:teuthology.task.internal:Duration was 1175.039345 seconds 2023-11-24T06:09:58.311 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-24T06:09:58.322 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-24T06:09:58.322 DEBUG:teuthology.orchestra.run.smithi091:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-24T06:09:58.325 DEBUG:teuthology.orchestra.run.smithi094:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-24T06:09:58.369 INFO:teuthology.orchestra.run.smithi091.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-24T06:09:58.371 INFO:teuthology.orchestra.run.smithi094.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-24T06:09:58.631 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-24T06:09:58.632 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi091.front.sepia.ceph.com 2023-11-24T06:09:58.632 DEBUG:teuthology.orchestra.run.smithi091:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-24T06:09:58.652 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi094.front.sepia.ceph.com 2023-11-24T06:09:58.652 DEBUG:teuthology.orchestra.run.smithi094:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-24T06:09:58.674 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-24T06:09:58.674 DEBUG:teuthology.orchestra.run.smithi091:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-24T06:09:58.695 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-24T06:09:58.754 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-24T06:09:58.755 DEBUG:teuthology.orchestra.run.smithi091:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-24T06:09:58.777 DEBUG:teuthology.orchestra.run.smithi094:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-24T06:09:58.869 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-24T06:09:58.915 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-24T06:09:58.915 DEBUG:teuthology.orchestra.run.smithi091:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-24T06:09:58.942 DEBUG:teuthology.orchestra.run.smithi094:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-24T06:09:58.969 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-24T06:09:58.983 DEBUG:teuthology.orchestra.run.smithi091:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-24T06:09:58.985 DEBUG:teuthology.orchestra.run.smithi094:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-24T06:09:59.008 INFO:teuthology.orchestra.run.smithi091.stdout:kernel.core_pattern = core 2023-11-24T06:09:59.035 INFO:teuthology.orchestra.run.smithi094.stdout:kernel.core_pattern = core 2023-11-24T06:09:59.049 DEBUG:teuthology.orchestra.run.smithi091:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-24T06:09:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:09:59.078 DEBUG:teuthology.orchestra.run.smithi094:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-24T06:09:59.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-24T06:09:59.105 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-24T06:09:59.115 INFO:teuthology.task.internal:Transferring archived files... 2023-11-24T06:09:59.116 DEBUG:teuthology.misc:Transferring archived files from smithi091:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547/remote/smithi091 2023-11-24T06:09:59.116 DEBUG:teuthology.orchestra.run.smithi091:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-24T06:09:59.155 DEBUG:teuthology.misc:Transferring archived files from smithi094:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547/remote/smithi094 2023-11-24T06:09:59.156 DEBUG:teuthology.orchestra.run.smithi094:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-24T06:09:59.193 INFO:teuthology.task.internal:Removing archive directory... 2023-11-24T06:09:59.193 DEBUG:teuthology.orchestra.run.smithi091:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-24T06:09:59.195 DEBUG:teuthology.orchestra.run.smithi094:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-24T06:09:59.243 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-24T06:09:59.254 INFO:teuthology.task.internal:Not uploading archives. 2023-11-24T06:09:59.254 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-24T06:09:59.265 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-24T06:09:59.265 DEBUG:teuthology.orchestra.run.smithi091:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-24T06:09:59.267 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-24T06:09:59.281 INFO:teuthology.orchestra.run.smithi091.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 24 06:09 /home/ubuntu/cephtest 2023-11-24T06:09:59.300 INFO:teuthology.orchestra.run.smithi094.stdout: 262320 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 24 06:09 /home/ubuntu/cephtest 2023-11-24T06:09:59.301 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-24T06:09:59.313 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-24T06:09:59.367 INFO:teuthology.nuke:Checking targets against current locks 2023-11-24T06:09:59.395 DEBUG:teuthology.nuke:shortname: smithi091 2023-11-24T06:09:59.395 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-24T06:09:59.426 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi091.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-24 05:45:14.982474', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-24T06:09:59.428 DEBUG:teuthology.nuke:shortname: smithi094 2023-11-24T06:09:59.428 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-24T06:09:59.451 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi094.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2023-11-24_05:41:33-fs:volumes-rishabh-mgr-vol-misc-testing-default-smithi/7465547', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-24 05:45:14.983959', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-24T06:09:59.480 INFO:teuthology.orchestra.console.smithi091:Power off 2023-11-24T06:09:59.480 DEBUG:teuthology.orchestra.console.smithi091:pexpect command: ipmitool -H smithi091.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-24T06:09:59.502 DEBUG:teuthology.orchestra.console.smithi091:power off output: Chassis Power Control: Down/Off 2023-11-24T06:09:59.502 DEBUG:teuthology.orchestra.console.smithi091:pexpect command: ipmitool -H smithi091.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T06:09:59.521 INFO:teuthology.orchestra.console.smithi094:Power off 2023-11-24T06:09:59.522 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-24T06:09:59.531 DEBUG:teuthology.orchestra.console.smithi091:check power output: Chassis Power is on 2023-11-24T06:09:59.546 DEBUG:teuthology.orchestra.console.smithi094:power off output: Chassis Power Control: Down/Off 2023-11-24T06:09:59.546 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T06:09:59.569 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2023-11-24T06:09:59.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-11-24T06:09:59.587 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-11-24T06:09:59.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-11-24T06:10:03.531 DEBUG:teuthology.orchestra.console.smithi091:pexpect command: ipmitool -H smithi091.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T06:10:03.571 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T06:10:03.642 DEBUG:teuthology.orchestra.console.smithi091:check power output: Chassis Power is on 2023-11-24T06:10:03.682 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is on 2023-11-24T06:10:07.646 DEBUG:teuthology.orchestra.console.smithi091:pexpect command: ipmitool -H smithi091.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T06:10:07.683 DEBUG:teuthology.orchestra.console.smithi094:pexpect command: ipmitool -H smithi094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T06:10:07.757 DEBUG:teuthology.orchestra.console.smithi091:check power output: Chassis Power is on 2023-11-24T06:10:07.792 DEBUG:teuthology.orchestra.console.smithi094:check power output: Chassis Power is off 2023-11-24T06:10:07.892 INFO:teuthology.orchestra.console.smithi094:Power off completed 2023-11-24T06:10:11.760 DEBUG:teuthology.orchestra.console.smithi091:pexpect command: ipmitool -H smithi091.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-24T06:10:11.870 DEBUG:teuthology.orchestra.console.smithi091:check power output: Chassis Power is off 2023-11-24T06:10:11.971 INFO:teuthology.orchestra.console.smithi091:Power off completed 2023-11-24T06:10:12.076 INFO:teuthology.run:Summary data: description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} duration: 1175.0393450260162 failure_reason: 'Test failure: test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e29d0b327c4647df8c3c77dcdd0507f6 status: fail success: false 2023-11-24T06:10:12.076 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-24T06:10:12.152 INFO:teuthology.run:FAIL